# springboot整合oss **Repository Path**: overload__hcf/File-Upload ## Basic Information - **Project Name**: springboot整合oss - **Description**: No description available - **Primary Language**: Java - **License**: AGPL-3.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 1 - **Created**: 2021-11-28 - **Last Updated**: 2025-02-20 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # SpringBoot集成oss ## 1.技术栈 SpringBoot MySQL Mybatis Shiro oss ## 2.使用shiro进行登录认证 > 有关文件操作的接口需要先登录认证才能访问 ![image-20211130181137039](https://gitee.com/overload__hcf/File-Upload/raw/master/%E9%A1%B9%E7%9B%AE%E6%88%AA%E5%9B%BE/image-20211130181137039.png) > 自定义ShiroConfig类 > > 1.注入安全管理器 DefaultWebSecurityManager > > 2.注入授权认证realm对象 UserRealm > > 3.配置ShiroFilterFactoryBean,设置内置过滤器以及安全管理器 ```java @Configuration public class ShiroConfig { @Bean(name = "webSecurityManager") public DefaultWebSecurityManager getDefaultWebSecurityManager(@Qualifier("userRealm") UserRealm userRealm){...} @Bean public UserRealm userRealm(){...} @Bean public ShiroFilterFactoryBean getShiroFilterFactoryBean( @Qualifier("webSecurityManager") DefaultWebSecurityManager defaultWebSecurityManager) {...} } ``` > 在controller层调用subject.login(UsernamePasswordToken),将会执行Userrealm中的认证方法 ```java UsernamePasswordToken token = new UsernamePasswordToken(username, DigestUtils.md5DigestAsHex(password.getBytes())); //执行登入方法，如果没有异常则执行成功 try { subject.login(token); return "redirect:/file/showAll"; } {...} ``` UserRealm ```java //认证 @Override protected AuthenticationInfo doGetAuthenticationInfo(AuthenticationToken authenticationToken) throws AuthenticationException { UsernamePasswordToken userToken = (UsernamePasswordToken) authenticationToken; User user = userService.findUserByName(userToken.getUsername()); if (user == null){ return null; } Subject subject = SecurityUtils.getSubject(); Session session = subject.getSession(); session.setAttribute("user", user); //密码认证,密码错误则会抛出 //IncorrectCredentialsException异常,被controller层处理 return new SimpleAuthenticationInfo(user,user.getPassword(),this.getName()); } ``` ## 3.使用oss进行文件上传 > 采取的是比较简易的方案,通过服务器做中转,浏览器上传是会先上传在服务器的一个临时目录中,然后通过秘钥buckname等信息连接oss后再将 文件进行上传.且采取MD5方式进行上传完整性校验.并将文件信息(旧名,新名,url等)存储在数据库中 ```java /** * 如果上传文件时设置了Content-MD5，OSS会根据接收的内容计算MD5。 * OSS计算的MD5值和上传提供的MD5值不一致时，则返回InvalidDigest异常，从而保证数据的完整性。 * 返回InvalidDigest异常后，您需要重新上传文件。 */ @Override public void oosUploadFile(MultipartFile multipartFile, Integer user_id) throws OSSException,IOException{ OSS ossClient = null; try { // 创建OSSClient实例。 ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret); //获取文件信息 UserFile userFile = SaveInUserFile(multipartFile, user_id); //设置objectName为上传的路径加文件名 String objectName = userFile.getPath() + "/" + userFile.getNewFileName(); ObjectMetadata meta = new ObjectMetadata(); //将字节流转换为字节数组 byte[] bytes = IOUtils.toByteArray(multipartFile.getInputStream()); // 设置MD5校验。 String md5 = BinaryUtil.toBase64String(BinaryUtil.calculateMd5(bytes)); log.info("md5============="+md5); meta.setContentMD5(md5); //进行提交 ossClient.putObject(bucketName, objectName,new ByteArrayInputStream(bytes),meta); //将文件信息保存到数据库 fileMapper.SaveFile(userFile); } finally { // 关闭OSSClient。 ossClient.shutdown(); } } ``` ![image-20211130183402730](https://gitee.com/overload__hcf/File-Upload/raw/master/%E9%A1%B9%E7%9B%AE%E6%88%AA%E5%9B%BE/image-20211130183402730.png) ## 4.使用oss进行文件下载 > 也是通过服务器进行中转,先连接oss客户端,根据数据库中文件的路径信息下载对应文件,将文件读入Servlet响应输出流中.读取后进行crc校验,判读是否文件完整. ```java public void ossDownloadFile(String objectName, HttpServletResponse response) throws OSSException,IOException { log.info("ossDownloadFile() called with parameters => 【objectName = {}】", objectName); OSS ossClient = null; try { // 创建OSSClient实例。 ossClient = new OSSClientBuilder().build(endpoint, accessKeyId, accessKeySecret); //流式下载 GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName, objectName); // 调用ossClient.getObject返回一个OSSObject实例，该实例包含文件内容及文件元信息。 OSSObject ossObject = ossClient.getObject(bucketName, objectName); // 调用ossObject.getObjectContent获取文件输入流，可读取此输入流获取其内容。 InputStream content = ossObject.getObjectContent(); //获取响应输出流 ServletOutputStream os = null; try { os = response.getOutputStream(); IOUtils.copy(content,os); } finally { IOUtils.closeQuietly(content); IOUtils.closeQuietly(os); } // 查看客户端是否开启了crc校验，默认是开启状态。 Boolean isCrcCheckEnabled = ((OSSClient)ossClient).getClientConfiguration().isCrcCheckEnabled(); log.info("isCrcCheckEnabled========"+isCrcCheckEnabled); // 查看是否是范围下载请求。范围下载方式不支持校验crc。 Boolean isRangGetRequest = getObjectRequest.getHeaders().get(OSSHeaders.RANGE) != null; log.info("isRangGetRequest========="+isRangGetRequest); // 校验crc，只有读取文件内容之后才能获取clientCrc。 if (isCrcCheckEnabled && !isRangGetRequest) { Long clientCRC = com.aliyun.oss.common.utils.IOUtils.getCRCValue(ossObject.getObjectContent()); OSSUtils.checkChecksum(clientCRC, ossObject.getServerCRC(), ossObject.getRequestId()); } //文件拷贝 } finally { // 关闭OSSClient。 ossClient.shutdown(); } ``` ## 5.将数据库两百多万条数据进行导出 > 在数据库中插入百万条数据 存储过程+insert ...select ```mysql drop procedure if exists authors_func; DELIMITER // CREATE PROCEDURE authors_func ( ) BEGIN DECLARE i INT; SET i = 1; SET @MIN = '2020-01-01 00:00:01'; SET @MAX = '2030-12-31 23:59:59'; WHILE i <= 10000 DO INSERT INTO `demo`.`authors` ( `first_name`, `last_name`, `email`, `birthdate`, `added` ) VALUES ( ( SELECT concat( substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', rand( ) * 26+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 26+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 26+1, 1 ) ) ), ( SELECT concat( substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', rand( ) * 26+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 52+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 26+1, 1 ), substring( 'abcdefghijklmnopqrstuvwxyz', rand( ) * 26+1, 1 ) ) ), ( SELECT concat( substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 62+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 62+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 72+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 72+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 72+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 72+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 72+1, 1 ), substring( 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890', rand( ) * 62+1, 1 ), '@example.net' ) ), ( SELECT NOW( ) - INTERVAL FLOOR( RAND( ) * 14 ) DAY ), ( SELECT TIMESTAMPADD( SECOND, FLOOR( RAND( ) * TIMESTAMPDIFF( SECOND, @MIN, @MAX ) ), @MIN ) ) ); SET i = i + 1; END WHILE; END;// DELIMITER ; ``` 先得到1万条数据,再执行8遍以下sql,每次执行都会复制一倍的数据,最后得到200万条数据 ```mysql insert authors(first_name, last_name, email, birthdate, added) select first_name, last_name, email, birthdate, added from authors ``` > 数据导出: > > 1.为了避免OOM不能将全量数据一次性加载到内存之中,要实现分批加载 > > 2.Mysql本身支持Stream查询，我们可以通过Stream流获取数据，然后将数据逐条刷入到文件中，每次刷入文件后再从内存中移除这条数据，从而避免OOM。 > > 3.通过mybatis实现,fetch+自定义ResultHandler ​ 自定义ResultHandler ```java public class CustomResultHandler implements ResultHandler { private final HttpServletResponse response; //封装response public CustomResultHandler(HttpServletResponse response) { super(); this.response = response; String fileName = System.currentTimeMillis() + ".csv"; this.response.addHeader("Content-Type", "application/csv"); this.response.addHeader("Content-Disposition", "attachment; filename="+fileName); this.response.setCharacterEncoding("UTF-8"); } @Override public void handleResult(ResultContext resultContext) { // 这里获取流式查询每次返回的单条结果 Authors authors = (Authors)resultContext.getResultObject(); processData(authors); } public void processData(E record) { try { //如果是要写入csv,需要重写toString,属性通过","分割 response.getWriter().write(record.toString()); response.getWriter().write("\n"); }catch (IOException e){ e.printStackTrace(); } } } ``` sql语句 ```xml ``` service实现,基于sqlsessionTemplates ```java public void streamDownload(HttpServletResponse httpServletResponse) throws IOException { CustomResultHandler customResultHandler = new CustomResultHandler(httpServletResponse); sqlSessionTemplate.select( "com.overload.fileupldwn.mapper.AuthorsMapper.streamByExample", customResultHandler); httpServletResponse.getWriter().flush(); httpServletResponse.getWriter().close(); } ``` > 文件开始下载 ![image-20211128190933656](https://gitee.com/overload__hcf/File-Upload/raw/master/%E9%A1%B9%E7%9B%AE%E6%88%AA%E5%9B%BE/image-20211128190933656.png) > 内存信息,总体比较稳定,没有明显的内存上涨趋势 ![image-20211128190924999](https://gitee.com/overload__hcf/File-Upload/raw/master/%E9%A1%B9%E7%9B%AE%E6%88%AA%E5%9B%BE/image-20211128190924999.png) > 下载完成后,数据完整 ![image-20211128193254541](https://gitee.com/overload__hcf/File-Upload/raw/master/%E9%A1%B9%E7%9B%AE%E6%88%AA%E5%9B%BE/image-20211128193254541.png) > 缺点:1.下载速度慢,fetchSize看网上是设置成-2147483648,我尝试改成1000会发现请求会被挂起,不懂为啥. > > 2.缺少完整性校验 ## 6.待优化(正在开发) 由于之前文件上传都是基于服务器做中转站的,继续开发通过客户端直传的方式,通过回调保存文件信息.减少服务器负载.