CentOS Server Notes — Commands, Tomcat OOM, Apache 403, Fonts, SSH

These were posted one at a time while running CentOS servers in 2015 and 2016, now collected onto a single page. Each was two or three lines, which made them harder to find than to read.

Everything here is CentOS 6. It is a record of what blocked me and how I got past it, not advice for a server today.

Commands I used constantly

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# find where a program is installed
which java
/usr/bin/java

# find and delete only old log files
find /logs/httpd/ -type f -mtime +180 -exec rm -f {} \;

# edit hosts
vi /etc/hosts

# edit the DNS server
vi /etc/resolv.conf

# check the OS bit width
getconf LONG_BIT

# check the OS version
cat /etc/issue

# list fonts
fc-list

# register fonts (after dropping them into /usr/share/fonts)
fc-cache -fv

# tail a log
tail -f file.log -n 1000

# crond
# list
crontab -l
# edit
crontab -e 

# check the tomcat process
ps -ef | grep tomcat

# kill a process
kill -9 {PID}

find ... -mtime +180 -exec rm got the most use. Logs piling up and filling the disk was a recurring problem.

Frequent OutOfMemory in Tomcat

Add this at the top of catalina.sh to raise the memory. Size it against the actual server memory.

1
export CATALINA_OPTS="-Djava.awt.headless=true -server -Xms2048m -Xmx2048m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"

PermGen was usually the part that blew up — hence MaxPermSize and CMSClassUnloadingEnabled.

Reference http://stackoverflow.com/questions/88235/dealing-with-java-lang-outofmemoryerror-permgen-space-error

Moving MySQL / MariaDB to utf8mb4

Storing emoji does not work with utf8. MySQL’s utf8 only holds three bytes.

1. Check the current state

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
MariaDB [(none)]> show variables like "%character%";show variables like "%collation%";
 +--------------------------+---------------------------------+
 | Variable_name            | Value                           |
 +--------------------------+---------------------------------+
 | character_set_client     | utf8                            |
 | character_set_connection | utf8                            |
 | character_set_database   | utf8                            |
 | character_set_filesystem | binary                          |
 | character_set_results    | utf8                            |
 | character_set_server     | utf8                            |
 | character_set_system     | utf8                            |
 | character_sets_dir       | /usr/local/mysql/share/charsets/|
 +--------------------------+---------------------------------+

2. Edit /etc/my.cnf

1
2
3
4
5
6
7
8
9
[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4

[mysqld]
collation-server = utf8mb4_unicode_ci
character-set-server = utf8mb4

3. Restart

1
service mysql restart

4. Verify

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
MariaDB [(none)]> show variables like "%character%";show variables like "%collation%";

+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)

+----------------------+--------------------+
| Variable_name        | Value              |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database   | utf8mb4_unicode_ci |
| collation_server     | utf8mb4_unicode_ci |
+----------------------+--------------------+
3 rows in set (0.00 sec)

character_set_system stays utf8. That is expected.

This is not the end of it. Changing the server settings is not enough — the columns themselves have to be converted to utf8mb4 too. Skip that and the configuration changes while the data still breaks.

Bypassing the cache while tuning queries

Running the same query a second time gets faster from the cache, which hides the real cost. Run this first and the numbers reflect an uncached execution.

1
RESET QUERY CACHE;

Apache returning 403 no matter what

When the configuration is correct and 403 persists, it is usually the SELinux context.

1
chcon -R --reference=/var/www /www/webroot

1
chcon -R -h -t httpd_sys_content_t /www/webroot

The first copies the context from /var/www; the second sets the type directly.

jfreeChart rendering Korean as boxes

Running jfreeChart under Tomcat renders Korean text as ㅁㅁㅁ, because the server has no matching font.

Drop the font file into /usr/share/fonts and run fc-cache -fv.

1
2
3
4
5
6
7
8
// specify the font in code
private static final Font _BASE_FONT = new Font("나눔고딕",Font.PLAIN,11);
...
(omitted)
'''
// use setFont where appropriate
lineAndShapeRenderer.setLegendTextFont(i, _BASE_FONT);
...

No Tomcat restart needed.

EC2 blocks ssh password access by default. This was on CentOS 6.5.

1
2
# set a password on the root account
passwd root

1
2
3
4
5
6
7
8
sudo vi /etc/ssh/sshd_config

# set these two options to yes, then save and close
PermitRootLogin yes
PasswordAuthentication yes

# restart sshd
service sshd restart

This is for convenience, not a setting to recommend. The key-based login below is the right answer.

Set up a key on macOS so logging into the server needs no password. ~~Normally I leave this off for security.~~

Generate the key on macOS

1
ssh-keygen -t rsa -C "name"

Print the public key.

1
cat ~/.ssh/id_rsa.pub

Register it on the target server (CentOS)

Add the public key from above to ~/.ssh/authorized_keys.

1
vi ~/.ssh/authorized_keys

Summary

Clean up old logs with find -mtime +N -exec rm
Tomcat OOM is usually PermGen — raise MaxPermSize
utf8mb4 needs both the server settings and the column types changed
Correct config but still 403 means SELinux — use chcon
Korean rendering as boxes means a missing font — install it and run fc-cache -fv
Prefer SSH keys over opening up password authentication