作者 | 蔡柱梁
審校 | 重樓
1.前言
在這個(gè)月的某個(gè)項(xiàng)目中,我們面臨了一項(xiàng)重要任務(wù),即每年一次的等保測(cè)評(píng)整改。這次測(cè)評(píng)的重點(diǎn)是MySQL的一些高危漏洞,我們被要求解決這些漏洞。因此,我們決定研究一下MySQL5.7升級(jí)到MySQL8.0的過(guò)程。在本文的最后,我將分享一些在這次升級(jí)過(guò)程中遇到的問(wèn)題。
升級(jí)需求:將5.7.43升級(jí)到8.0.34, 升級(jí)方式 in-place升級(jí)【關(guān)閉現(xiàn)有版本MySQL,將二進(jìn)制或包替換成新版本并在現(xiàn)有數(shù)據(jù)目錄上啟動(dòng)MySQL并執(zhí)行升級(jí)任務(wù)的方式,稱為in-place升級(jí)】
原版本 | 5.7.43 | CentOS Linux release 7.9.2009 |
新版本 | 8.0.34 | CentOS Linux release 7.9.2009 |
2.MySQL生命周期
以下MySQL生命周期-內(nèi)容來(lái)自于互聯(lián)網(wǎng)
從Oracle發(fā)布的版本生命周期規(guī)劃可以看到,MySQL 5.7已經(jīng)走到了生命周期的終點(diǎn),意味著后續(xù)將不再為MySQL 5.7提供官方更新、錯(cuò)誤修復(fù)或安全補(bǔ)丁。
3.MySQL 8.0的新特性
- 默認(rèn)字符集由latin1變?yōu)閡tf8mb4
- MyISAM系統(tǒng)表全部換成InnoDB表
- JSON特性增強(qiáng)
- 支持不可見索引,支持直方圖
- sql_mode參數(shù)默認(rèn)值變化
- 默認(rèn)密碼策略變更
- 新增角色管理
- 支持窗口函數(shù),支持Hash join
4.升級(jí)建議
支持從MySQL 5.7升級(jí)到MySQL 8.0,注意僅支持GA版本之間的升級(jí)。
不支持跨大版本的升級(jí),如從5.6升級(jí)到8.0是不支持的。
建議升級(jí)大版本前先升級(jí)到當(dāng)前版本的最近小版本,如5.7先升級(jí)到5.7.43后再升級(jí)到8.0。
做好充足的備份!數(shù)據(jù)無(wú)價(jià)!
5.升級(jí)前準(zhǔn)備
5.1 MySQL-shell 檢查工具兼容性
在執(zhí)行升級(jí)操作前需要做一些檢查工作,確認(rèn)準(zhǔn)備工作是否就緒,避免升級(jí)過(guò)程中出現(xiàn)異常。MySQL Shell使用util.checkForServerUpgrade進(jìn)行檢查,返回內(nèi)容包括不符合遷移要求的問(wèn)題,error的問(wèn)題需要遷移前修改。
MySQL-shell 下載地址:https://dev.mysql.com/downloads/shell/
選擇 Archives ,查詢更多版本。
選擇當(dāng)前最新的版本8.0.34,x84,64-bit 。
#下載包
[root@srebro.cn ~]# wget https://downloads.mysql.com/archives/get/p/43/file/mysql-shell-8.0.34-linux-glibc2.12-x86-64bit.tar.gz -C /root
[root@srebro.cn ~]# tar -xf mysql-shell-8.0.34-linux-glibc2.12-x86-64bit.tar.gz
[root@srebro.cn ~]# cd /root/mysql-shell-8.0.34-linux-glibc2.12-x86-64bit/bin
[root@srebro.cn bin]# ./mysqlsh -uroot -p -S /tmp/mysql.sock -e "util.checkForServerUpgrade()" > util.checkForServerUpgrade.log
輸出報(bào)告
The MySQL server at /tmp%2Fmysql.sock, version 5.7.39-log - MySQL Community
Server (GPL), will now be checked for compatibility issues for upgrade to MySQL
8.0.34...
1) Usage of old temporal type
No issues found
2) MySQL 8.0 syntax check for routine-like objects
No issues found
3) Usage of db objects with names conflicting with new reserved keywords
No issues found
4) Usage of utf8mb3 charset
Warning: The following objects use the utf8mb3 character set. It is
recommended to convert them to use utf8mb4 instead, for improved Unicode
support.
More information:
https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html
mysql - schema's default character set: utf8
test - schema's default character set: utf8
5) Table names in the mysql schema conflicting with new tables in 8.0
No issues found
6) Partitioned tables using engines with non native partitioning
No issues found
7) Foreign key constraint names longer than 64 characters
No issues found
8) Usage of obsolete MAXDB sql_mode flag
No issues found
9) Usage of obsolete sql_mode flags
Notice: The following DB objects have obsolete options persisted for
sql_mode, which will be cleared during upgrade to 8.0.
More information:
https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html#mysql-nutshell-removals
global system variable sql_mode - defined using obsolete NO_AUTO_CREATE_USER
option
10) ENUM/SET column definitions containing elements longer than 255 characters
No issues found
11) Usage of partitioned tables in shared tablespaces
No issues found
12) Circular directory references in tablespace data file paths
No issues found
13) Usage of removed functions
No issues found
14) Usage of removed GROUP BY ASC/DESC syntax
No issues found
15) Removed system variables for error logging to the system log configuration
To run this check requires full path to MySQL server configuration file to be specified at 'configPath' key of options dictionary
More information:
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-13.html#mysqld-8-0-13-logging
16) Removed system variables
To run this check requires full path to MySQL server configuration file to be specified at 'configPath' key of options dictionary
More information:
https://dev.mysql.com/doc/refman/8.0/en/added-deprecated-removed.html#optvars-removed
17) System variables with new default values
To run this check requires full path to MySQL server configuration file to be specified at 'configPath' key of options dictionary
More information:
https://mysqlserverteam.com/new-defaults-in-mysql-8-0/
18) Zero Date, Datetime, and Timestamp values
No issues found
19) Schema inconsistencies resulting from file removal or corruption
No issues found
20) Tables recognized by InnoDB that belong to a different engine
No issues found
21) Issues reported by 'check table x for upgrade' command
No issues found
22) New default authentication plugin considerations
Warning: The new default authentication plugin 'caching_sha2_password' offers
more secure password hashing than previously used 'mysql_native_password'
(and consequent improved client connection authentication). However, it also
has compatibility implications that may affect existing MySQL installations.
If your MySQL installation must serve pre-8.0 clients and you encounter
compatibility issues after upgrading, the simplest way to address those
issues is to reconfigure the server to revert to the previous default
authentication plugin (mysql_native_password). For example, use these lines
in the server option file:
[mysqld]
default_authentication_plugin=mysql_native_password
However, the setting should be viewed as temporary, not as a long term or
permanent solution, because it causes new accounts created with the setting
in effect to forego the improved authentication security.
If you are using replication please take time to understand how the
authentication plugin changes may impact you.
More information:
https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-caching-sha2-password-compatibility-issues
https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html#upgrade-caching-sha2-password-replication
23) Columns which cannot have default values
No issues found
24) Check for invalid table names and schema names used in 5.7
No issues found
25) Check for orphaned routines in 5.7
No issues found
26) Check for deprecated usage of single dollar signs in object names
No issues found
27) Check for indexes that are too large to work on higher versions of MySQL
Server than 5.7
No issues found
28) Check for deprecated '.<table>' syntax used in routines.
No issues found
Errors: 0
Warnings: 3
Notices: 1
NOTE: No fatal errors were found that would prevent an upgrade, but some potential issues were detected. Please ensure that the reported issues are not significant before upgrading.
從輸出報(bào)告可以看出,升級(jí)檢查器在28個(gè)方面進(jìn)行了檢查,最終得出3個(gè)警告信息和1個(gè)提示。
消除警告:
Usage of utf8mb3 charset 在 MySQL 8.0版本之前,默認(rèn)字符集為latin1 ,utf8字符集指向的是utf8mb3 。從MySQL8.0開始,數(shù)據(jù)庫(kù)的默認(rèn)編碼將改為utf8mb4 ;為了避免新舊對(duì)象字符集不一致的情況,可以在配置文件將字符集和校驗(yàn)規(guī)則設(shè)置為舊版本的字符集和比較規(guī)則。
New default authentication plugin considerations,密碼認(rèn)證插件變更。為了避免連接問(wèn)題,可以仍采用5.7的mysql_native_password認(rèn)證插件。
消除提示:
Usage of obsolete sql_mode flags:MySQL 8.0 版本sql_mode不支持NO_AUTO_CREATE_USER,要避免配置的sql_mode中帶有NO_AUTO_CREATE_USER。
通過(guò)以上的例子,可以發(fā)現(xiàn),MySQL Shell提供的升級(jí)檢查工具能夠幫助我們檢測(cè)版本兼容性,減輕升級(jí)工作負(fù)擔(dān)。
5.2 邏輯備份MySQL數(shù)據(jù)
which mysqldump
/home/application/mysql/app/bin/mysqldump
# --routines 備份存儲(chǔ)過(guò)程和函數(shù);--set-gtid-purged=OFF: 禁用GTID(全局事務(wù)標(biāo)識(shí));xxx1,XXX2 表示庫(kù)名,備份多個(gè)庫(kù)用空格做為間隔
/home/application/mysql/app/bin/mysqldump -uroot -p --routines --set-gtid-purged=OFF --databases XXX1 XXX2 > /root/all-database-20231026.sql
5.3 優(yōu)雅的停止數(shù)據(jù)庫(kù)
# 進(jìn)入原5.7 mysql命令行 正確關(guān)閉數(shù)據(jù)庫(kù)
[root@srebro.cn ~]# mysql -uroot -p'srebro'
mysql> select version();
+------------+
| version() |
+------------+
| 5.7.43-log |
+------------+
1 row in set (0.00 sec)
mysql> show variables like 'innodb_fast_shutdown';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| innodb_fast_shutdown | 1 |
+----------------------+-------+
1 row in set (0.00 sec)
# 確保數(shù)據(jù)都刷到硬盤上,更改成0
InnoDB 關(guān)閉模式。
如果值為 0,InnoDB 會(huì)在關(guān)閉前進(jìn)行緩慢關(guān)閉、完全清除和更改緩沖區(qū)合并。
如果值為 1(默認(rèn)值),InnoDB 會(huì)在關(guān)閉時(shí)跳過(guò)這些操作,這個(gè)過(guò)程稱為快速關(guān)閉。
如果值為 2,InnoDB 刷新其日志并冷關(guān)機(jī),就好像 MySQL 崩潰了;沒有提交的事務(wù)丟失,但崩潰恢復(fù)操作使下一次啟動(dòng)需要更長(zhǎng)的時(shí)間。在仍然緩沖大量數(shù)據(jù)的極端情況下,緩慢關(guān)閉可能需要幾分鐘甚至幾小時(shí)。
mysql> set global innodb_fast_shutdown=0;
Query OK, 0 rows affected (0.00 sec)
mysql> shutdown;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
[root@cmdb ~]# ps -ef | grep mysql
root 30990 30934 0 16:12 pts/0 00:00:00 grep --color=auto mysql
5.4 備份MySQL 數(shù)據(jù)目錄,安裝目錄和配置文件
--確認(rèn)數(shù)據(jù)庫(kù)狀態(tài)為關(guān)閉狀態(tài)
[root@srebro.cn ~]# systemctl status mysqld
--數(shù)據(jù)目錄備份
[root@srebro.cn ~]# cp -r /home/application/mysql/data /home/application/mysql/data_bak_`date +%F`
--安裝目錄備份
[root@srebro.cn ~]# cp -r /home/application/mysql/app/ /home/application/mysql/app_bak_`date +%F`
--配置文件備份
[root@srebro.cn ~]# cp /etc/my.cnf /etc/my.cnf_`date +%F`
5.5 下載并解壓MySQL8.0
https://dev.mysql.com/downloads/
選擇 Archives ,查詢更多版本。
下載地址:https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.34-linux-glibc2.12-x86_64.tar.xz 選擇mysql-8.0.34-linux-glibc2.12-x86_64.tar.xz。
#執(zhí)行以下步驟解壓tar包:
# 安裝包上傳至原安裝包目錄下 我的是/home/application/mysql
[root@srebro.cn ~]# cd /home/application/mysql
[root@srebro.cn mysql]# wget https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.34-linux-glibc2.12-x86_64.tar.xz
[root@srebro.cn mysql]# tar -xf mysql-8.0.34-linux-glibc2.12-x86_64.tar.xz
# 文件夾重命名為mysql8
[root@srebro.cn mysql]# mv mysql-8.0.34-linux-glibc2.12-x86_64 mysql8
# 更改文件夾所屬
[root@srebro.cn mysql]# chown -Rf mysql:mysql /home/application/mysql/mysql8
# 刪除安裝包
[root@srebro.cn mysql]# rm -rf mysql-8.0.34-linux-glibc2.12-x86_64.tar.xz
6.升級(jí)
6.1 修改 my.cnf 配置文件
因5.7版本與8.0版本參數(shù)有所不同,為了能順利升級(jí),我們需要更改部分配置參數(shù)。主要注意sql_mode、basedir、密碼認(rèn)證插件及字符集設(shè)置,其他參數(shù)最好還是按照原5.7的來(lái),不需要做調(diào)整。下面展示5.7和8.0的配置文件,注意備份原來(lái)配置文件。
6.1.1 MySQL5.7_my.cnf 配置文件
[mysql]
socket=/tmp/mysql.sock
default-character-set=utf8
[mysqld]
user=mysql
basedir=/home/application/mysql/app
datadir=/home/application/mysql/data
character_set_server=utf8
collation-server=utf8_general_ci
#日志時(shí)間
log_timestamps=SYSTEM
port=3306
socket=/tmp/mysql.sock
max_connections=1000
max_allowed_packet=500M
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
#慢日志
long_query_time=3
slow_query_log=ON
slow_query_log_file=/home/application/mysql/slow_query.log
#錯(cuò)誤日志
log-error=/home/application/mysql/mysql-error.log
#binlog配置
server_id=150
log-bin=mysql-bin
max_binlog_size = 100M
binlog_format=row
log_slave_updates
expire_logs_days=7
#只能用IP地址檢查客戶端的登錄,不用主機(jī)名
skip-name-resolve=1
6.1.2 MySQL8.0_my.cnf 配置文件
[mysql]
socket=/tmp/mysql.sock
default-character-set=utf8
[mysqld]
user=mysql
#日志時(shí)間
log_timestamps=SYSTEM
port=3306
socket=/tmp/mysql.sock
max_connections=1000
max_allowed_packet=500M
#只能用IP地址檢查客戶端的登錄,不用主機(jī)名
skip-name-resolve=1
#binlog配置
server_id=150
log-bin=mysql-bin
max_binlog_size = 100M
binlog_format=row
log_slave_updates
expire_logs_days=7
#慢日志
long_query_time=3
slow_query_log=ON
slow_query_log_file=/home/application/mysql/slow_query.log
#錯(cuò)誤日志
log-error=/home/application/mysql/mysql-error.log
#for8.0
sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
basedir=/home/application/mysql/mysql8
datadir=/home/application/mysql/data
character_set_server=utf8
collation-server=utf8_general_ci
# 默認(rèn)使用"mysql_native_password"插件認(rèn)證
default_authentication_plugin=mysql_native_password
# 創(chuàng)建新表時(shí)將使用的默認(rèn)存儲(chǔ)引擎
default-storage-engine=INNODB
6.2 執(zhí)行升級(jí)程序
在MySQL 5.7升級(jí)的時(shí)候,MySQL啟動(dòng)后還需執(zhí)行mysql_upgrade后重啟MySQL。MySQL8.0.16開始,MySQL 不推薦使用mysql_upgrade;直接使用 mysqld_safe 直接啟動(dòng)。
關(guān)于--upgrade=的一些參數(shù):
--upgrade=AUTO MySQL升級(jí)所有過(guò)時(shí)的內(nèi)容。
--upgrade=NONE MySQL跳過(guò)升級(jí)步驟,可能會(huì)導(dǎo)致報(bào)錯(cuò)。
--upgrade=MINIMAL MySQL在必要時(shí)升級(jí)數(shù)據(jù)字典表,information_schema和information_schema。這可能會(huì)導(dǎo)致部分功能不能正常使用,例如MGR 。
--upgrade=FORCE MySQL會(huì)升級(jí)所有的內(nèi)容,這會(huì)檢查所有schema的所有對(duì)象,導(dǎo)致MySQL需要更長(zhǎng)的時(shí)間啟動(dòng)。此模式下,MySQL會(huì)重新創(chuàng)建系統(tǒng)表if they are missing。
[root@srebro.cn ~]# /home/application/mysql/mysql8/bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql --upgrade=FORCE
2023-10-30T07:33:23.595626Z mysqld_safe Logging to '/home/application/mysql/mysql-error.log'.
2023-10-30T07:33:23.620303Z mysqld_safe Starting mysqld daemon with databases from /home/application/mysql/data
如出現(xiàn)一直卡住,不用擔(dān)心!
新開一個(gè)窗口,可觀察下錯(cuò)誤日志看是否報(bào)錯(cuò)/home/application/mysql/mysql-error.log 然后登錄數(shù)據(jù)庫(kù)測(cè)試。
[root@srebro.cn ~]# mysql -uroot -p'srebro'
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.34 MySQL Community Server - GPL
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select version();
+-----------+
| version() |
+-----------+
| 8.0.34 |
+-----------+
1 row in set (0.00 sec)
mysql>
6.3 修改MySQL環(huán)境變量
由于basedir 從/home/application/mysql/app 變成了 /home/application/mysql/mysql8,需要修改下環(huán)境變量信息:
[root@srebro.cn ~]# vim /etc/profile
export PATH=$PATH:/home/application/mysql/mysql8/bin
.......
#使環(huán)境變量生效
[root@srebro.cn ~]# source /etc/profile
#驗(yàn)證下mysql環(huán)境變量
#退出當(dāng)前終端
[root@srebro.cn ~]# exit
[root@srebro.cn ~]# which mysql
/home/application/mysql/mysql8/bin/mysql
[root@srebro.cn ~]# mysql -V
mysql Ver 8.0.34 for Linux on x86_64 (MySQL Community Server - GPL)
6.4 停止mysqld_safe進(jìn)程,使用systemd管理MySQL 8.0
[root@cmdb ~]# kill -9 `ps -ef | grep mysql | awk '{print $2}'`
#確認(rèn)沒有mysql進(jìn)程
[root@cmdb ~]# ps -ef | grep mysql
#使用systemd管理mysql8
#修改原先的ExecStart中,basedir的路徑,改為mysql8 的路徑
[root@cmdb ~]# vim /etc/systemd/system/mysqld.service
[Unit]
Descriptinotallow=MySQL Server
Documentatinotallow=man:mysqld
Documentatinotallow=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/home/application/mysql/mysql8/bin/mysqld --defaults-file=/etc/my.cnf
6.5 配置MySQL 8.0開機(jī)自啟&啟動(dòng)MySQL 8.0
#reload下systemd
[root@cmdb ~]# systemctl daemon-reload
#加入開機(jī)自啟動(dòng)
[root@cmdb ~]# systemctl enable mysqld
Created symlink from /etc/systemd/system/multi-user.target.wants/mysqld.service to /etc/systemd/system/mysqld.service.
#啟動(dòng)mysql8數(shù)據(jù)庫(kù)
[root@cmdb ~]# systemctl start mysqld
[root@cmdb ~]# ps -ef | grep mysql
mysql 9497 1 36 14:59 ? 00:00:01 /home/application/mysql/mysql8/bin/mysqld --defaults-file=/etc/my.cnf
root 9544 8560 0 14:59 pts/0 00:00:00 grep --color=auto mysql
#登錄數(shù)據(jù)庫(kù)驗(yàn)證
[root@cmdb ~]# mysql -uroot -p'srebro'
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.34 MySQL Community Server - GPL
Copyright (c) 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select version();
+-----------+
| version() |
+-----------+
| 8.0.34 |
+-----------+
1 row in set (0.00 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| srebro |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql>
7.遇到的問(wèn)題
7.1 問(wèn)題一
在升級(jí)MySQL8.0后,關(guān)于JDBC中SSL連接的一些報(bào)錯(cuò)信息,如下圖:
經(jīng)排查發(fā)現(xiàn),MySQL 8.0數(shù)據(jù)庫(kù)默認(rèn)開啟了SSL認(rèn)證,且之前MySQL 5.7.39 也是默認(rèn)開啟了SSL認(rèn)證。代碼和JDBC驅(qū)動(dòng)版本都沒有變化,那很有可能就是MySQL 8.0 中對(duì)于SSL的一個(gè)變化。咨詢了DBA 朋友,專業(yè)的解釋是,在5.7.31的時(shí)候SSL在源碼中貌似沒有真正的起作用,后面版本完善了這塊的內(nèi)容。倘若,不使用SSL去連接,就需要按照如下的方法去處理:
- 從數(shù)據(jù)庫(kù)方面,直接在my.cnf 中 添加skip_ssl 參數(shù),從源頭上關(guān)閉SSL 認(rèn)證的方式
- 從代碼層面,在JDBC 連接中,使用 &useSSL=false 參數(shù),表示不使用SSL 認(rèn)證
7.2 問(wèn)題二
MySQL報(bào)錯(cuò)unblock with ‘mysqladmin flush-hosts’,報(bào)錯(cuò)如下:
JDBC連接報(bào)錯(cuò),報(bào)錯(cuò)內(nèi)容 ERROR 1129 (HY000): Host '192.168.1.34' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
原因:同一個(gè)ip在短時(shí)間內(nèi)產(chǎn)生太多,中斷的數(shù)據(jù)庫(kù)連接而導(dǎo)致的阻塞;而中斷的原因是有些業(yè)務(wù)使用SSL去連接數(shù)據(jù)庫(kù),導(dǎo)致登錄失敗,登錄被鎖。
臨時(shí)解決方法:使用mysqladmin flush-hosts 命令清理一下hosts文件,mysqladmin -u xxx -p flush-hosts,從根本上去解決,就需要排查什么異常的連接導(dǎo)致阻塞,登錄被鎖,比如上面提到的SSL認(rèn)證的問(wèn)題。
參考資料
- https://www.modb.pro/db/1715541568826990592
- https://www.modb.pro/db/530848
- https://www.modb.pro/db/1716302208709517312
作者介紹
蔡柱梁,51CTO社區(qū)編輯,從事Java后端開發(fā)8年,做過(guò)傳統(tǒng)項(xiàng)目廣電BOSS系統(tǒng),后投身互聯(lián)網(wǎng)電商,負(fù)責(zé)過(guò)訂單,TMS,中間件等。