SpringSpring 4.0 Integration Notes — JUnit, XML View, RabbitMQ, Redis, Batch, SiteMesh, Hadoop
2015 · 11 · 10
6 min read
Paper
This collects eight short configuration notes posted through 2015 while wiring various
things into a Spring 4.0 project. Individually they were fragments that were harder to
find than to read.
The order is by usefulness rather than by date. The JUnit setup comes first because
everything else was verified through it.
All versions are from 2015. This is a record of what was wired up and how, not something
to copy into a project today.
JUnit test cases
Everything below is verified through this, so it goes first.
1
2
3
4
5
6
| <dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>4.0.5.RELEASE</version>
<scope>test</scope>
</dependency>
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| @RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration( {
"classpath:servlet-context.xml",
"classpath:config/context-datasource.xml"
}
)
public class MemberServiceTest {
@Autowired
MemberService memberService;
@Test
public void testSr2002() throws Exception {
RequestData req = new RequestData(null, new DbMap());
ResponseData res = new ResponseData(new DbMap());
memberService.sr2002(req, res);
}
}
|
The problem case is a DB obtained through a jndi-lookup — there is no JNDI in tests.
Adding test/resources/config/context-datasource.xml overrides the existing bean id.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource"
p:driverClassName="com.mysql.jdbc.Driver"
p:url="jdbc:mysql://server:3306/dbname"
p:username="sa"
p:password="" />
</beans>
|
With the same id, whichever loads later wins. This pattern is reused in the integration
tests below.
An XML ViewResolver for Map results
Everything had been returning JSON until a situation came up that needed XML. This view
class only applies when the result is a Map.
applicationServlet.xml
1
2
3
4
| <beans:bean id="xmlView2" class="org.springframework.web.servlet.view.XmlViewResolver">
<beans:property name="order" value="1"/>
<beans:property name="location" value="classpath:xml-views.xml"/>
</beans:bean>
|
xml-views.xml
1
2
3
4
5
6
7
8
9
10
| <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN"
"http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<bean name="xmlView" class="com.xxxxx.view.AjaxResponseXMLView">
<property name="contentType">
<value>text/xml;charset=utf-8</value>
</property>
</bean>
</beans>
|
AjaxResponseXMLView.java
It walks Maps and Lists recursively, emitting tags as it goes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
| public class AjaxResponseXMLView extends AbstractView {
@Override
protected void renderMergedOutputModel(Map map, HttpServletRequest request,
HttpServletResponse response) throws Exception {
String xmlHeader = "\r\n";
StringBuffer xmlSb = new StringBuffer();
xmlSb.append(xmlHeader);
xmlSb.append("");
writeFromMap(xmlSb, map);
xmlSb.append("");
response.setContentType("application/xml");
response.setCharacterEncoding("utf-8");
response.setHeader("Cache-Control", "no-cache");
response.setContentLength(xmlSb.toString().getBytes("utf-8").length);
response.getWriter().print(xmlSb.toString());
}
private void writeFromMap(StringBuffer sb, Map map) {
for(Object str : map.keySet()) {
Object v = map.get(str);
sb.append("<" + str + ">");
if(v instanceof Map) {
writeFromMap(sb, (Map) v);
}
else if(v instanceof List) {
writeFromList(sb, (List) v);
}
else {
writeFromData(sb, v);
}
sb.append("</" + str + ">");
}
}
private void writeFromList(StringBuffer sb, List list) {
for(Object v : list) {
sb.append("");
if(v instanceof Map) {
writeFromMap(sb, (Map)v);
}
else if(v instanceof List) {
writeFromList(sb, (List) v);
}
else {
writeFromData(sb, v);
}
sb.append("");
}
}
private void writeFromData(StringBuffer sb, Object data) {
sb.append(escapeXml(data+""));
}
private String escapeXml(String src) {
src = src.replace("\"", """);
src = src.replace("<", "<");
src = src.replace(">", ">");
src = src.replace("&", "&");
return src;
}
}
|
RabbitMQ
Installed straight from an rpm.
1
2
3
4
5
| # start the server
sbin/rabbitmq-server start
# stop the server
sbin/rabbitmqctl stop
|
pom.xml
1
2
3
4
5
| <dependency>
<groupid>org.springframework.amqp</groupId>
<artifactid>spring-rabbit</artifactId>
<version>1.4.1.RELEASE</version>
</dependency>
|
context-rabbitmq.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <!-- A reference to the org.springframework.amqp.rabbit.connection.ConnectionFactory -->
<rabbit:connection-factory id="connectionFactory" host="localhost" username="worker" password="workerpassword" />
<!-- Creates a org.springframework.amqp.rabbit.core.RabbitTemplate for access to the broker -->
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" />
<!-- Creates a org.springframework.amqp.rabbit.core.RabbitAdmin to manage exchanges, queues and bindings -->
<rabbit:admin connection-factory="connectionFactory" />
<!-- Creates a queue for consumers to retrieve messages -->
<rabbit:queue name="simple_queue" />
<rabbit:listener-container connection-factory="connectionFactory">
<rabbit:listener queues="simple_queue" ref="mqService" />
</rabbit:listener-container>
|
MqService.java
Both the sending and receiving sides live in one class. Implementing MessageListener
is enough for the listener-container above to deliver messages to it.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| @Service
public class MqService implements MessageListener {
private static final Logger logger = LoggerFactory.getLogger(MqService.class);
private static final String TASK_QUEUE_NAME = "simple_queue";
@Autowired
private RabbitTemplate rabbitTemplate;
public void send(String message) throws IOException {
rabbitTemplate.convertAndSend(TASK_QUEUE_NAME, message);
logger.info("send message={}", message);
}
@Override
public void onMessage(Message message) {
String msg = null;
try {
msg = new String(message.getBody(), "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
logger.info("recv message=" + msg );
}
}
|
Redis
pom.xml
1
2
3
4
5
6
7
8
9
10
| <dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>1.1.0.RELEASE</version>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.1.0</version>
</dependency>
|
The versions have to match. Otherwise some classes go missing and it fails.
context-redis.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:p="http://www.springframework.org/schema/p"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">
<bean id="jedisConnFactory"
class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory"
p:usePool="true"
p:hostName="172.xxx.xxx.xxx"
p:port="6379"
/>
<!-- redis template definition -->
<bean id="redisTemplate"
class="org.springframework.data.redis.core.RedisTemplate"
p:connectionFactory-ref="jedisConnFactory"
/>
</beans>
|
6379 is the redis default port. It can be changed at install time.
RedisTest.java
ValueOperations sets the value together with an expiry.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| @RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration( {
"classpath:spring/application-context.xml"
}
)
public class RedisTest {
private static final Logger logger = LoggerFactory.getLogger(RedisTest.class);
@Autowired
RedisTemplate<String, String> redisTemplate;
@Resource(name="redisTemplate")
private ValueOperations<String, ResultMap> valueOps;
@Test
public void testTp4110() throws Exception {
// redisTemplate.delete("1");
ResultMap res = valueOps.get("1");
if(res == null) {
logger.info("create.. cache..");
// create..
// 10 minute cache
valueOps.set("1", ResultMap.create(), 10, TimeUnit.MINUTES);
res = valueOps.get("1");
}
logger.info("redis-test={}", res);
res = valueOps.get("2");
logger.info("redis-test={}", res);
}
}
|
The output. A missing key (“2”) comes back as null.
1
2
3
4
5
| ...
[INFO ] 17:30:28.990 [main] - create.. cache..
[INFO ] 17:30:29.190 [main] - redis-test={result_code=0000, result_message=success}
[INFO ] 17:30:29.288 [main] - redis-test=null
...
|
Spring Batch
This worked out well. The nice part is that the transaction commit size and the read
size can be set independently.
A job, summarized:
the reader reads data, the processor handles it, the writer records the result.
It also provides listeners that fire at each stage boundary. The reader and writer here
are not custom — they are the ones mybatis ships
(see https://mybatis.github.io/spring/ko/batch.html).
The configuration mattered more than the queries or logic, so that is what is kept here.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
| <bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
<property name="jobRepository" ref="jobRepository" />
</bean>
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager"/>
</bean>
<!-- sent type sent -->
<job:job id="rstJob" job-repository="jobRepository">
<job:step id="step1">
<tasklet>
<chunk reader="rstReader" processor="memberRstProcess"
writer="rstWriter" commit-interval="500">
</chunk>
</tasklet>
</job:step>
</job:job>
<bean id="memberRstProcess" class="com.xxxxx.MemberRstProcess" />
<bean id="rstReader"
class="org.mybatis.spring.batch.MyBatisPagingItemReader"
p:sqlSessionFactory-ref="sqlSessionFactory"
p:queryId="com.xxxxx.mapper.QueryMapper.selectMemberRstList"
p:pageSize="500"
scope="step" />
<bean id="rstWriter" class="org.mybatis.spring.batch.MyBatisBatchItemWriter">
<property name="sqlSessionFactory" ref="sqlSessionFactory" />
<property name="statementId" value="com.xxxxx.mapper.QueryMapper.updateMemberRst" />
</bean>
|
pageSize is the read unit and commit-interval is the commit unit. They are separate.
SiteMesh
pom.xml
1
2
3
4
5
| <dependency>
<groupId>opensymphony</groupId>
<artifactId>sitemesh</artifactId>
<version>2.4.2</version>
</dependency>
|
WEB-INF/web.xml
1
2
3
4
5
6
7
8
| <filter>
<filter-name>sitemesh</filter-name>
<filter-class>com.opensymphony.module.sitemesh.filter.PageFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>sitemesh</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
|
WEB-INF/sitemesh.xml
There is almost nothing to change in this file beyond the decorators.xml path.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| <?xml version="1.0" encoding="UTF-8"?>
<sitemesh>
<property name="decorators-file" value="/WEB-INF/decorators.xml" />
<excludes file="${decorators-file}" />
<page-parsers>
<parser content-type="text/html"
class="com.opensymphony.module.sitemesh.parser.HTMLPageParser" />
<parser content-type="text/html;charset=UTF-8"
class="com.opensymphony.module.sitemesh.parser.HTMLPageParser" />
</page-parsers>
<decorator-mappers>
<mapper class="com.opensymphony.module.sitemesh.mapper.PrintableDecoratorMapper">
<param name="decorator" value="printable" />
<param name="parameter.name" value="printable" />
<param name="parameter.value" value="true" />
</mapper>
<mapper class="com.opensymphony.module.sitemesh.mapper.PageDecoratorMapper" >
<param name="property" value="meta.decorator" />
</mapper>
<mapper class="com.opensymphony.module.sitemesh.mapper.ConfigDecoratorMapper">
<param name="config" value="${decorators-file}" />
</mapper>
</decorator-mappers>
</sitemesh>
|
WEB-INF/decorators.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <?xml version="1.0" encoding="UTF-8"?>
<decorators defaultdir="/decorators">
<excludes>
<pattern>/*.json</pattern>
</excludes>
<decorator name="top" page="/views/layout/top.jsp" />
<decorator name="left" page="/views/layout/left.jsp" />
<decorator name="layout2" page="/views/layout/layout2.jsp">
<pattern>/login</pattern>
<pattern>/login_error</pattern>
</decorator>
<decorator name="layout" page="/views/layout/layout.jsp">
<pattern>/*</pattern>
</decorator>
</decorators>
|
Dropping in the XML is the whole setup.
<excludes> takes URL patterns that should not get a decorator<decorator> is the actual jsp layout or template- It has a name and a page; the name can be applied from another decorator via
<page:applyDecorator name="top" /> <pattern>/login</pattern> specifies which URLs the decorator applies to
Building the layout
The base layout.jsp. This is a trimmed summary rather than the file actually in use.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
| <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="decorator" uri="http://www.opensymphony.com/sitemesh/decorator"%>
<%@ taglib prefix="page" uri="http://www.opensymphony.com/sitemesh/page" %>
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>sample</title>
<decorator:head/>
</head>
<body>
<div class="wrapper">
<header class="main-header">
<page:applyDecorator name="top" />
</header>
<!-- Left side column. contains the logo and sidebar -->
<aside class="main-sidebar">
<page:applyDecorator name="left" />
</aside>
<!-- Content Wrapper. Contains page content -->
<div class="content-wrapper">
<decorator:body />
</div><!-- /.content-wrapper -->
<footer class="main-footer">
<div class="pull-right hidden-xs">
<b>Version</b> 0.0.1
</div>
<strong>Copyright © 2015 sample </strong> All rights reserved.
</footer>
</div>
</body>
</html>
|
<decorator:head /> pulls in the <head> content of the target page<page:applyDecorator name="top" /> pulls in the top decorator — think of it as an include<page:applyDecorator name="left" /> does the same for left<decorator:body /> pulls in the <body> content of the target page
An actual MVC jsp looks like this.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
| <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<head>
<!-- page script -->
<script>
....
</script>
</head>
<body>
<!-- Content Header (Page header) -->
<section class="content-header">
<h1>
데이타 관리
<small>버전관리</small>
</h1>
<ol class="breadcrumb">
<li><a href="#"><i class="fa fa-dashboard"></i> Home</a></li>
<li><a href="#">데이타관리</a></li>
<li class="active">버전관리</li>
</ol>
</section>
<!-- Main content -->
<section class="content">
<div class="row">
<div class="col-xs-12">
<div class="box">
<div class="box-header">
<h3 class="box-title">버전 목록</h3>
<a href="create"><button type="button" class="btn btn-primary btn-lg pull-right">신규 추가</button></a>
</div><!-- /.box-header -->
<div class="box-body">
<table id="list" class="table table-bordered table-hover">
</table>
</div><!-- /.box-body -->
</div><!-- /.box -->
</div><!-- /.col -->
</div><!-- /.row -->
</section><!-- /.content -->
</body>
|
Giving any jsp just a <head> and a <body> renders it inside layout.jsp.
Hadoop
Installed 2.6.x on OSX Yosemite.
pom.xml
1
2
3
4
5
| <dependency>
<groupid>org.springframework.data</groupId>
<artifactid>spring-data-hadoop</artifactId>
<version>2.1.1.RELEASE</version>
</dependency>
|
context-hadoop.xml
1
2
3
4
5
6
7
8
9
10
11
| <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:hdp="http://www.springframework.org/schema/hadoop"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">
<hdp:configuration id="hdConf">
fs.default.name=hdfs://localhost:9000
</hdp:configuration>
</beans>
|
Reading and writing a file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
| @RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({
"classpath:servlet-context.xml",
"classpath:config/context-datasource.xml",
"classpath:config/context-hadoop.xml"
})
public class HdTestServiceTest {
private static final Logger logger = LoggerFactory.getLogger(HdTestService.class);
@Autowired
private org.apache.hadoop.conf.Configuration hdConf;
@Test
public void testDoTest() throws Exception {
FileSystem hdfs = null;
try {
Path filePath = new Path("/tmp/test.txt");
logger.info("filePath.uri={}", filePath.toUri());
hdfs = FileSystem.get(filePath.toUri(), hdConf);
if(hdfs.exists(filePath)) {
logger.info("read file path={}", filePath);
BufferedReader r = new BufferedReader(new InputStreamReader(hdfs.open(filePath), "utf-8"));
String line = null;
do {
line = r.readLine();
logger.info(" line={}", line);
}
while(line != null);
r.close();
// dfs.delete(filePath, true);
} else {
logger.info("create new file path={}", filePath);
FSDataOutputStream out = hdfs.create(filePath, false);
out.write("한글 생성 테스트".getBytes("utf-8"));
out.flush();
out.close();
}
}
finally {
IOUtils.closeQuietly(hdfs);
}
}
}
|
It works, though still only locally. There is no server to move it to.
This much is enough to implement file upload and download.
Hadoop MapReduce — WordCount
A job gets added to the configuration above.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <hdp:configuration id="hdConf">
fs.default.name=hdfs://localhost:9000
</hdp:configuration>
<hdp:job id="wordCountJob"
input-path="/input/"
output-path="/output/"
configuration-ref="hdConf"
mapper="delim.app.service.WordCount$TokenizerMapper"
reducer="delim.app.service.WordCount$IntSumReducer"
>
</hdp:job>
<hdp:job-runner id="wordCountJobRunner" job-ref="wordCountJob" run-at-startup="false">
</hdp:job-runner>
|
WordCount.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
| import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.StringTokenizer;
public class WordCount {
private static final Logger logger = LoggerFactory.getLogger(WordCount.class);
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
@Override
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
logger.info("map key={}, value={}", key, value);
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
logger.info("reduce key={}", key);
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
}
|
Running it
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| @Autowired
private org.apache.hadoop.conf.Configuration hdConf;
@Autowired
private JobRunner wordCountJobRunner;
@Before
public void beforeCopyFile() throws IOException {
String file = "/Users/paper/Desktop/4/14/debug.2015-04-09.log";
Path srcFilePath = new Path(file);
Path dstFilePath = new Path("/input/debug.2015-04-09.log");
FileSystem hdfs = FileSystem.get(dstFilePath.toUri(), hdConf);
hdfs.copyFromLocalFile(false, true, srcFilePath, dstFilePath);
hdfs.delete(new Path("/output/"), true);
}
@Test
public void testRunJob() throws Exception {
wordCountJobRunner.call();
}
|
The sequence:
@Before copies the local debug.log into hdfs- The job runs
- You can see
debug.log being read line by line (WordCount$TokenizerMapper)
Summary
- Start integration testing from the JUnit setup. Override a JNDI datasource by reusing
the same bean id in a test-only xml
- For RabbitMQ, attaching a
MessageListener implementation to listener-container is all it takes - For Redis, the jedis and spring-data-redis versions must match or classes go missing
- In Spring Batch, read size (
pageSize) and commit size (commit-interval) are independent - SiteMesh needs three XML files: web.xml, sitemesh.xml, decorators.xml
- For Hadoop, a single
hdp:configuration is enough to inject a usable FileSystem