-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathsearch.xml
322 lines (192 loc) · 162 KB
/
search.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
<?xml version="1.0" encoding="utf-8"?>
<search>
<entry>
<title>Spring boot 过滤器配置及源码调试</title>
<link href="/2018/12/27/Java/Spring%20boot-filter-configure-and-source-code-review/"/>
<url>/2018/12/27/Java/Spring%20boot-filter-configure-and-source-code-review/</url>
<content type="html"><![CDATA[<h2 id="自定义过滤器"><a href="#自定义过滤器" class="headerlink" title="自定义过滤器"></a>自定义过滤器</h2><ol><li>创建自定义拦截器类 <code>MyFilterA</code> 实现 <code>Filter</code> 接口</li><li>重写 <code>doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain)</code> 方法</li></ol><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">MyFilterA</span> <span class="keyword">implements</span> <span class="title">Filter</span> </span>{</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">doFilter</span><span class="params">(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain)</span> <span class="keyword">throws</span> IOException, ServletException </span>{</span><br><span class="line"> filterChain.doFilter(servletRequest,servletResponse);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="Java-代码配置过滤器"><a href="#Java-代码配置过滤器" class="headerlink" title="Java 代码配置过滤器"></a>Java 代码配置过滤器</h2><ol start="3"><li>创建配置类 <code>FilterConfiguration</code></li><li>使用注解 <code>@Configuration</code></li><li>定义一个 bean</li></ol><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Bean</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> FilterRegistrationBean <span class="title">customFilter</span><span class="params">()</span></span>{</span><br><span class="line"> FilterRegistrationBean registration = <span class="keyword">new</span> FilterRegistrationBean();</span><br><span class="line"> registration.setFilter(<span class="keyword">new</span> MyFilterA());</span><br><span class="line"> registration.addUrlPatterns(<span class="string">"/*"</span>);</span><br><span class="line"> registration.setName(<span class="string">"customFilter1"</span>);</span><br><span class="line"> registration.setOrder(<span class="number">1</span>);</span><br><span class="line"> <span class="keyword">return</span> registration;</span><br><span class="line">}</span><br></pre></td></tr></table></figure><h2 id="注解配置"><a href="#注解配置" class="headerlink" title="注解配置"></a>注解配置</h2><ul><li>最直接的方法就是在自定义过滤器类加上 <code>@Component</code> 注解，但是这种方式会过滤所有请求，而且没法配置</li><li>在自定义类添加 <code>@WebFilter(urlPatterns = "/*", filterName = "customFilter1")</code> 注解，并且在启动类上添加 <code>@ServletComponentScan</code> 注解</li></ul><h2 id="多个-Filter-过滤顺序"><a href="#多个-Filter-过滤顺序" class="headerlink" title="多个 Filter 过滤顺序"></a>多个 Filter 过滤顺序</h2><ul><li><p>代码配置<br>定义多个 CustomFilter ，在配置类中定义多个 Bean ，设置 <code>registration.setOrder(1);</code> 进行过滤器排序</p></li><li><p>注解配置<br>网上看到好多帖子里说要在过滤器上添加 <code>@Order()</code> 注解，经过测试这是无效的<br><a href="https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/core/annotation/Order.html" target="_blank" rel="noopener">JavaDoc:org.springframework.core.annotation.Order</a> 的描述是</p><blockquote><p>Since Spring 4.0, annotation-based ordering is supported for many kinds of components in Spring</p></blockquote><p> 支持部分组件的排序，而且不会影响单例的启动顺序，Filter 也是单例的吧。</p></li></ul><h2 id="结尾"><a href="#结尾" class="headerlink" title="结尾"></a>结尾</h2><p>以上就是 Spring Boot 关于过滤器的配置</p><h2 id="折腾"><a href="#折腾" class="headerlink" title="折腾"></a>折腾</h2><p>不折腾不舒服星人表示不知道原因就很不爽。<br>找到第请求时初始化过 <code>FilterChain</code> 的位置<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create the filter chain for this request</span></span><br><span class="line">ApplicationFilterChain filterChain =</span><br><span class="line"> ApplicationFilterFactory.createFilterChain(request, wrapper, servlet);</span><br></pre></td></tr></table></figure></p><p>创建 <code>createFilterChain</code> 方法</p><p>可以看到 <code>new ApplicationFilterChain();</code> 只是创建了一个空的 <code>ApplicationFilterChain</code> 对象</p><p>继续步进，发现 <code>FilterMap filterMaps[] = context.findFilterMaps();</code> 从 context 获取了 FilterMap 的数组</p><p>再往下开始遍历数组，匹配 url 规则，然后获取 <code>filterName</code> 通过 <code>filterName</code> 获取 <code>FilterConfig</code> 并调用 <code>filterChain.addFilter(filterConfig);</code> 添加到 <code>FilterChain</code> 本质上是内置一个名为 <code>filters</code> 的数组。</p><p>经过以上外部的步骤可以大概知道过滤器大概的工作流程了，但我这次是要找过滤器是如何排序的，所以现在看看 <code>context.findFilterMaps();</code> 如何获取 <code>filterMaps[]</code> 的。<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Construct a FilterChain implementation that will wrap the execution of</span></span><br><span class="line"><span class="comment"> * the specified servlet instance.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> request The servlet request we are processing</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> wrapper The wrapper managing the servlet instance</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> servlet The servlet instance to be wrapped</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> The configured FilterChain instance or null if none is to be</span></span><br><span class="line"><span class="comment"> * executed.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> ApplicationFilterChain <span class="title">createFilterChain</span><span class="params">(ServletRequest request,</span></span></span><br><span class="line"><span class="function"><span class="params"> Wrapper wrapper, Servlet servlet)</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">// If there is no servlet to execute, return null</span></span><br><span class="line"> <span class="keyword">if</span> (servlet == <span class="keyword">null</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">null</span>;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Create and initialize a filter chain object</span></span><br><span class="line"> ApplicationFilterChain filterChain = <span class="keyword">null</span>;</span><br><span class="line"> <span class="keyword">if</span> (request <span class="keyword">instanceof</span> Request) {</span><br><span class="line"> Request req = (Request) request;</span><br><span class="line"> <span class="keyword">if</span> (Globals.IS_SECURITY_ENABLED) {</span><br><span class="line"> <span class="comment">// Security: Do not recycle</span></span><br><span class="line"> filterChain = <span class="keyword">new</span> ApplicationFilterChain();</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> filterChain = (ApplicationFilterChain) req.getFilterChain();</span><br><span class="line"> <span class="keyword">if</span> (filterChain == <span class="keyword">null</span>) {</span><br><span class="line"> filterChain = <span class="keyword">new</span> ApplicationFilterChain();</span><br><span class="line"> req.setFilterChain(filterChain);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// Request dispatcher in use</span></span><br><span class="line"> filterChain = <span class="keyword">new</span> ApplicationFilterChain();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> filterChain.setServlet(servlet);</span><br><span class="line"> filterChain.setServletSupportsAsync(wrapper.isAsyncSupported());</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Acquire the filter mappings for this Context</span></span><br><span class="line"> StandardContext context = (StandardContext) wrapper.getParent();</span><br><span class="line"> FilterMap filterMaps[] = context.findFilterMaps();</span><br><span class="line"></span><br><span class="line"> <span class="comment">// If there are no filter mappings, we are done</span></span><br><span class="line"> <span class="keyword">if</span> ((filterMaps == <span class="keyword">null</span>) || (filterMaps.length == <span class="number">0</span>))</span><br><span class="line"> <span class="keyword">return</span> filterChain;</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Acquire the information we will need to match filter mappings</span></span><br><span class="line"> DispatcherType dispatcher =</span><br><span class="line"> (DispatcherType) request.getAttribute(Globals.DISPATCHER_TYPE_ATTR);</span><br><span class="line"></span><br><span class="line"> String requestPath = <span class="keyword">null</span>;</span><br><span class="line"> Object attribute = request.getAttribute(Globals.DISPATCHER_REQUEST_PATH_ATTR);</span><br><span class="line"> <span class="keyword">if</span> (attribute != <span class="keyword">null</span>){</span><br><span class="line"> requestPath = attribute.toString();</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> String servletName = wrapper.getName();</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Add the relevant path-mapped filters to this filter chain</span></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < filterMaps.length; i++) {</span><br><span class="line"> <span class="keyword">if</span> (!matchDispatcher(filterMaps[i] ,dispatcher)) {</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (!matchFiltersURL(filterMaps[i], requestPath))</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> ApplicationFilterConfig filterConfig = (ApplicationFilterConfig)</span><br><span class="line"> context.findFilterConfig(filterMaps[i].getFilterName());</span><br><span class="line"> <span class="keyword">if</span> (filterConfig == <span class="keyword">null</span>) {</span><br><span class="line"> <span class="comment">// FIXME - log configuration problem</span></span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> filterChain.addFilter(filterConfig);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Add filters that match on servlet name second</span></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < filterMaps.length; i++) {</span><br><span class="line"> <span class="keyword">if</span> (!matchDispatcher(filterMaps[i] ,dispatcher)) {</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (!matchFiltersServlet(filterMaps[i], servletName))</span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> ApplicationFilterConfig filterConfig = (ApplicationFilterConfig)</span><br><span class="line"> context.findFilterConfig(filterMaps[i].getFilterName());</span><br><span class="line"> <span class="keyword">if</span> (filterConfig == <span class="keyword">null</span>) {</span><br><span class="line"> <span class="comment">// FIXME - log configuration problem</span></span><br><span class="line"> <span class="keyword">continue</span>;</span><br><span class="line"> }</span><br><span class="line"> filterChain.addFilter(filterConfig);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Return the completed filter chain</span></span><br><span class="line"> <span class="keyword">return</span> filterChain;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span> the set of filter mappings for this Context.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="keyword">public</span> FilterMap[] findFilterMaps() {</span><br><span class="line"> <span class="keyword">return</span> filterMaps.asArray();</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>是从 <code>org.apache.catalina.core.StandardContext</code> 类中获取的 filterMaps 注释的描述顺序是按部署描述定义的顺序映射的，这个内部也是个数组。另外<br>filterMaps 类型是 ContextFilterMaps ，这是 StandardContext 类的一个内部静态类，除了构造方法外只有四个方法，分别是 add addBefore asArray remove，<br>这里主要 add 和 addBefore ，实际上是被 StandardContext 的 addFilterMap 和 addFilterMapBefore ， 通过 findUsage 可以发现实际是被 addMappingForUrlPatterns 调用了。然后找 addMappingForUrlPatterns 被调用的地方</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * The set of filter mappings for this application, in the order</span></span><br><span class="line"><span class="comment"> * they were defined in the deployment descriptor with additional mappings</span></span><br><span class="line"><span class="comment"> * added via the {<span class="doctag">@link</span> ServletContext} possibly both before and after those</span></span><br><span class="line"><span class="comment"> * defined in the deployment descriptor.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">final</span> ContextFilterMaps filterMaps = <span class="keyword">new</span> ContextFilterMaps();</span><br></pre></td></tr></table></figure><p>以下代码可以倒序就是启动容器的流程<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">addMappingForUrlPatterns</span><span class="params">(</span></span></span><br><span class="line"><span class="function"><span class="params"> EnumSet<DispatcherType> dispatcherTypes, <span class="keyword">boolean</span> isMatchAfter,</span></span></span><br><span class="line"><span class="function"><span class="params"> String... urlPatterns)</span> </span>{</span><br><span class="line"></span><br><span class="line"> FilterMap filterMap = <span class="keyword">new</span> FilterMap();</span><br><span class="line"></span><br><span class="line"> filterMap.setFilterName(filterDef.getFilterName());</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (dispatcherTypes != <span class="keyword">null</span>) {</span><br><span class="line"> <span class="keyword">for</span> (DispatcherType dispatcherType : dispatcherTypes) {</span><br><span class="line"> filterMap.setDispatcher(dispatcherType.name());</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (urlPatterns != <span class="keyword">null</span>) {</span><br><span class="line"> <span class="comment">// % decoded (if necessary) using UTF-8</span></span><br><span class="line"> <span class="keyword">for</span> (String urlPattern : urlPatterns) {</span><br><span class="line"> filterMap.addURLPattern(urlPattern);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (isMatchAfter) {</span><br><span class="line"> context.addFilterMap(filterMap);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> context.addFilterMapBefore(filterMap);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// else error?</span></span><br><span class="line">}</span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Configure registration settings. Subclasses can override this method to perform</span></span><br><span class="line"><span class="comment"> * additional configuration if required.</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@param</span> registration the registration</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">configure</span><span class="params">(FilterRegistration.Dynamic registration)</span> </span>{</span><br><span class="line"> <span class="keyword">super</span>.configure(registration);</span><br><span class="line"> EnumSet<DispatcherType> dispatcherTypes = <span class="keyword">this</span>.dispatcherTypes;</span><br><span class="line"> <span class="keyword">if</span> (dispatcherTypes == <span class="keyword">null</span>) {</span><br><span class="line"> dispatcherTypes = EnumSet.of(DispatcherType.REQUEST);</span><br><span class="line"> }</span><br><span class="line"> Set<String> servletNames = <span class="keyword">new</span> LinkedHashSet<>();</span><br><span class="line"> <span class="keyword">for</span> (ServletRegistrationBean<?> servletRegistrationBean : <span class="keyword">this</span>.servletRegistrationBeans) {</span><br><span class="line"> servletNames.add(servletRegistrationBean.getServletName());</span><br><span class="line"> }</span><br><span class="line"> servletNames.addAll(<span class="keyword">this</span>.servletNames);</span><br><span class="line"> <span class="keyword">if</span> (servletNames.isEmpty() && <span class="keyword">this</span>.urlPatterns.isEmpty()) {</span><br><span class="line"> registration.addMappingForUrlPatterns(dispatcherTypes, <span class="keyword">this</span>.matchAfter,</span><br><span class="line"> DEFAULT_URL_MAPPINGS);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">if</span> (!servletNames.isEmpty()) {</span><br><span class="line"> registration.addMappingForServletNames(dispatcherTypes, <span class="keyword">this</span>.matchAfter,</span><br><span class="line"> StringUtils.toStringArray(servletNames));</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (!<span class="keyword">this</span>.urlPatterns.isEmpty()) {</span><br><span class="line"> registration.addMappingForUrlPatterns(dispatcherTypes, <span class="keyword">this</span>.matchAfter,</span><br><span class="line"> StringUtils.toStringArray(<span class="keyword">this</span>.urlPatterns));</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">protected</span> <span class="keyword">final</span> <span class="keyword">void</span> <span class="title">register</span><span class="params">(String description, ServletContext servletContext)</span> </span>{</span><br><span class="line"> D registration = addRegistration(description, servletContext);</span><br><span class="line"> <span class="keyword">if</span> (registration == <span class="keyword">null</span>) {</span><br><span class="line"> logger.info(StringUtils.capitalize(description) + <span class="string">" was not registered "</span></span><br><span class="line"> + <span class="string">"(possibly already registered?)"</span>);</span><br><span class="line"> <span class="keyword">return</span>;</span><br><span class="line"> }</span><br><span class="line"> configure(registration); <span class="comment">//调用configure</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">final</span> <span class="keyword">void</span> <span class="title">onStartup</span><span class="params">(ServletContext servletContext)</span> <span class="keyword">throws</span> ServletException </span>{</span><br><span class="line"> String description = getDescription();</span><br><span class="line"> <span class="keyword">if</span> (!isEnabled()) {</span><br><span class="line"> logger.info(StringUtils.capitalize(description)</span><br><span class="line"> + <span class="string">" was not registered (disabled)"</span>);</span><br><span class="line"> <span class="keyword">return</span>;</span><br><span class="line"> }</span><br><span class="line"> register(description, servletContext); <span class="comment">//调用register</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">private</span> <span class="keyword">void</span> <span class="title">selfInitialize</span><span class="params">(ServletContext servletContext)</span> <span class="keyword">throws</span> ServletException </span>{</span><br><span class="line"> prepareWebApplicationContext(servletContext);</span><br><span class="line"> registerApplicationScope(servletContext);</span><br><span class="line"> WebApplicationContextUtils.registerEnvironmentBeans(getBeanFactory(),</span><br><span class="line"> servletContext);</span><br><span class="line"> <span class="keyword">for</span> (ServletContextInitializer beans : getServletContextInitializerBeans()) {</span><br><span class="line"> beans.onStartup(servletContext); <span class="comment">//调用onStartup</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p><p>启动应用 -> 初始化Bean -> 排序 -> 迭代 -> 注册到上下文 -> 添加到 <code>FilterMaps</code> </p><p>访问url -> 从上下文（context）获取（findFilterMaps）-> 过滤匹配url - 添加到 <code>FilterChain</code> -> <code>doFilter()</code></p><p>看到这里，总算有了和排序相关的一些内容，篇幅有点长，不打算在这里继续往下写了，因为需要看 <code>BeanFactory</code> 如何从 <code>scanner</code> 注册 <code>bean</code> ，篇幅也很大。目前至少能知道 <code>Filter</code> 的顺序是在初始化 <code>context</code> 的时候就已经根据 <code>order</code> 值从小到大排好序了，<code>order</code> 的值在使用 <code>FilterRegistrationBean</code> 注册的时候是 <code>registration.setOrder(1);</code> 值，如果使用或不使用 <code>@Order</code> 注解 <code>order</code> 值都是 <code>2147483647</code></p><p>改天闲的没事干再看如何注册 Bean 吧，看的头都晕了</p>]]></content>
<tags>
<tag> java </tag>
</tags>
</entry>
<entry>
<title>Spring boot 关于过滤器与拦截器地址的尝试</title>
<link href="/2018/12/26/Java/Spring%20boot-about-url-path/"/>
<url>/2018/12/26/Java/Spring%20boot-about-url-path/</url>
<content type="html"><![CDATA[<p>RequestMapping:</p><p> <code>/</code> , <code>/index</code> , <code>/ab</code></p><p>拦截器：</p><pre><code>设置为 / localhost:8080/ 有效localhost:8080/index 无效localhost:8080/ab 无效 设置为 /* localhost:8080/ 有效localhost:8080/index 有效localhost:8080/ab 有效设置为 /**localhost:8080/ 有效localhost:8080/index 有效localhost:8080/ab 有效</code></pre><p>过滤器：</p><pre><code>设置为 /localhost:8080/ 有效localhost:8080/index 无效设置为 /*localhost:8080/ 有效localhost:8080/index 有效设置为 /**localhost:8080/ 无效localhost:8080/index 无效localhost:8080/ab 无效</code></pre><p>通过测试可以得出 <code>/</code> 只拦截根路径， <code>/*</code> 回会拦截所有的路径，<code>/**</code> 在拦截器中和/*效果一样，在过滤器中会失效</p><p>如果访问一个不在 controller 中定义的路径，拦截器会执行两遍,第一次是访问的路径拦截，第二次是 `/error’ 拦截</p><p>如果访问存在的路径（浏览器），过滤器会执行两遍,第一次是访问的路径过滤，第二次是 <code>/favicon.ico</code></p><p>小结：<br>拦截器是基于面向切面（AOP）的，只能对 controller 进行拦截；<br>过滤器是基于 servlet ，所以能拦截静态资源；</p>]]></content>
<tags>
<tag> java </tag>
</tags>
</entry>
<entry>
<title>Spring boot 拦截器配置</title>
<link href="/2018/12/26/Java/Spring%20boot-interceptor-configure/"/>
<url>/2018/12/26/Java/Spring%20boot-interceptor-configure/</url>
<content type="html"><![CDATA[<h2 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h2><ol><li>创建自定义拦截器，实现 <code>HandlerInterceptor</code> 接口</li><li>使用 <code>@Component</code> 注解</li><li>重写方法 <code>preHandle</code> 、 <code>postHandle</code> 、 <code>afterCompletion</code></li></ol><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Component</span></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">MyIntercepter</span> <span class="keyword">implements</span> <span class="title">HandlerInterceptor</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">static</span> Logger log = LoggerFactory.getLogger(MyIntercepter.class);</span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">boolean</span> <span class="title">preHandle</span><span class="params">(HttpServletRequest request, HttpServletResponse response, Object handler)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">postHandle</span><span class="params">(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">afterCompletion</span><span class="params">(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex)</span> <span class="keyword">throws</span> Exception </span>{</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><ol start="4"><li>创建 Configuration 类 ， 实现 <code>WebMvcConfigurer</code> 接口</li><li>使用 <code>@Configuration</code> 注解</li><li>注入自定义拦截器</li></ol><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Autowired</span></span><br><span class="line"><span class="keyword">private</span> MyIntercepter myIntercepter;</span><br></pre></td></tr></table></figure><ol start="7"><li>重写 <code>addInterceptors</code> 方法</li><li>添加拦截器，设置规则</li></ol><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Override</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">addInterceptors</span><span class="params">(InterceptorRegistry registry)</span></span>{</span><br><span class="line"> registry.addInterceptor(myIntercepter).addPathPatterns(<span class="string">"/**"</span>).excludePathPatterns(<span class="string">"/failed"</span>);</span><br><span class="line">}</span><br></pre></td></tr></table></figure>]]></content>
<tags>
<tag> java </tag>
</tags>
</entry>
<entry>
<title>CDH 6.0.0 安装记录</title>
<link href="/2018/12/04/Java/cdh6.0-install-process/"/>
<url>/2018/12/04/Java/cdh6.0-install-process/</url>
<content type="html"><![CDATA[<p>最近又来了几个服务器，整了一套虚拟环境，需要在上面部署一套 cdh hadoop 环境。不想再去看 csdn 上文档搬运工写的帖子了，直接看官方文档排排坑。<br><a id="more"></a></p><h2 id="Before-install"><a href="#Before-install" class="headerlink" title="Before install"></a>Before install</h2><ol><li>设置SELINUX模式<br>执行 <code>getenforce</code><br>如果输出 <code>Permissive</code> 或 <code>Disabled</code> 进行下一步（关闭防火墙）。<br>如果为 <code>enforcing</code> 执行以下操作:<br><code>sudo vim /etc/selinux/config</code><br>修改为 <code>SELINUX=permissive</code><br><img src="./cdh6.0-install-process/selinux.png" alt=""></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 使设置生效</span><br><span class="line">setenforce 0</span><br></pre></td></tr></table></figure><p>等安装部署完后可以再改回去<br>执行 <code>setenforce 1</code></p><ol start="2"><li><p>关闭防火墙</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">sudo systemctl status firewalld.service</span><br><span class="line"><span class="meta">#</span> 关闭防火墙</span><br><span class="line">sudo systemctl stop firewalld.service</span><br><span class="line"><span class="meta">#</span> 禁用</span><br><span class="line">sudo systemctl disable firewalld.service</span><br></pre></td></tr></table></figure></li><li><p>开启NTP</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">sudo yum install ntp</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span>配置时间同步服务器，默认有</span><br><span class="line">sudo vim /etc/ntp.conf</span><br><span class="line"></span><br><span class="line">sudo systemctl start ntpd</span><br><span class="line">sudo systemctl enable ntpd</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span>同步时间</span><br><span class="line">sudo hwclock --systohc</span><br></pre></td></tr></table></figure></li><li><p>设置主机名</p></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">sudo vim /etc/hostname</span><br><span class="line">centosxxx</span><br><span class="line"></span><br><span class="line">sudo vim /etc/hosts</span><br><span class="line"></span><br><span class="line">172.17.13.xxx centosxxx </span><br><span class="line">172.17.13.xxx centosxxx</span><br><span class="line">172.17.13.xxx centosxxx</span><br><span class="line">172.17.13.xxx centosxxx</span><br></pre></td></tr></table></figure><p>集群中每台机器都重复此操作</p><h2 id="Install"><a href="#Install" class="headerlink" title="Install"></a>Install</h2><ol><li>创建用户</li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">adduser cdh</span><br><span class="line">pwsswd cdh</span><br></pre></td></tr></table></figure><p>添加sudo权限<br><code>vim /etc/sodoers</code><br>添加 <code>cdh ALL=(ALL) NOPASSWD:ALL</code></p><p><img src="./cdh6.0-install-process/sudoers.png" alt=""><br>集群中每台机器都重复此操作</p><ol start="2"><li>设置dns<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/NetworkManager/NetworkManager.conf</span><br><span class="line"></span><br><span class="line">dns=none</span><br></pre></td></tr></table></figure></li></ol><p><img src="./cdh6.0-install-process/networkmanager.png" alt=""><br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/resolv.conf</span><br><span class="line"></span><br><span class="line">nameserver 114.114.114.114</span><br></pre></td></tr></table></figure></p><p><img src="./cdh6.0-install-process/resolv.png" alt=""></p><ol start="3"><li><p>设置仓库地址</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">su cdh</span><br><span class="line">sudo rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera</span><br></pre></td></tr></table></figure></li><li><p>设置JAVA_HOME</p><blockquote><p>Requirements<br>The JDK must be 64-bit. Do not use a 32-bit JDK.<br>jdk 必须为64位<br>The installed JDK must be a supported version as documented in <a href="https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#java_requirements" target="_blank" rel="noopener">Java Requirements</a>.<br>必须是<a href="https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#java_requirements" target="_blank" rel="noopener">Java Requirements</a>文档中支持的版本<br>The same version of the Oracle JDK must be installed on each cluster host.<br>集群中的每台机器的oracle jdk版本都必须相同<br>The JDK must be installed at /usr/java/jdk-version.<br>jdk必须安装在<code>/usr/java/jdk-version</code>目录</p></blockquote></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">vim /etc/profile</span><br><span class="line"></span><br><span class="line">JAVA_HOME=/usr/java/jdk1.8</span><br></pre></td></tr></table></figure><p>集群中每台机器都重复此操作</p><ol start="5"><li><p>下载rpm包<br>从 <a href="https://www.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cm_6_version_download.html#cm_600_version_download" target="_blank" rel="noopener">Cloudera Manager 6.0.0 仓库地址</a> 选择要安装的版本 这里选6.0<br>选择 <a href="https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/" target="_blank" rel="noopener">redhat7仓库地址</a> 下载地址</p></li><li><p>安装cloudera-manager-server</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 这里需要先配置 `JAVA_HOME` 否则会报错</span><br><span class="line">rpm -i cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm</span><br><span class="line"><span class="meta">#</span> 这里需要使用yum来解决依赖问题</span><br><span class="line">yum -install cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm</span><br><span class="line"><span class="meta">#</span> 同上</span><br><span class="line">yum -install cloudera-manager-server-6.0.0-530873.el7.x86_64.rpm</span><br></pre></td></tr></table></figure></li><li><p>安装数据库<br>这里 <a href="https://dev.mysql.com/downloads/file/?id=480402" target="_blank" rel="noopener">下载</a> mysql5.6 rpm包</p></li></ol><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 查看是否安装了 mariadb 数据库</span><br><span class="line">rpm -qa | grep mariadb</span><br><span class="line"><span class="meta">#</span> 如果有 卸载先</span><br><span class="line">sudo rpm -e mariadb-libs-5.5.44-2.el7.centos.x86_64 --nodeps</span><br><span class="line"><span class="meta">#</span> 安装 mysql</span><br><span class="line">sudo yum install MySQL-server-5.6.42-1.el7.x86_64.rpm </span><br><span class="line"><span class="meta">#</span> 安装 mysql-shared-compat Cloudera文档标示必须安装</span><br><span class="line">sudo yum install MySQL-shared-compat-5.6.42-1.el7.x86_64.rpm</span><br><span class="line"><span class="meta">#</span> 安装 mysql 客户端</span><br><span class="line">sudo yum install MySQL-client-5.6.42-1.el7.x86_64.rpm </span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 安装成功后查看初始化密码</span><br><span class="line">sudo cat /root/.mysql_secret</span><br><span class="line">mysql -uroot -p</span><br><span class="line"></span><br><span class="line">SET PASSWORD = PASSWORD('123456');</span><br><span class="line">quit</span><br><span class="line"></span><br><span class="line">sudo cp /usr/share/mysql/my-default.cnf my.cnf</span><br><span class="line">sudo vim /usr/share/mysql/my.cnf</span><br></pre></td></tr></table></figure><p>my.cnf<br><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">[mysqld]</span></span><br><span class="line"><span class="attr">datadir</span>=/var/lib/mysql</span><br><span class="line"><span class="attr">socket</span>=/var/lib/mysql/mysql.sock</span><br><span class="line"><span class="attr">transaction-isolation</span> = READ-COMMITTED</span><br><span class="line"><span class="comment"># Disabling symbolic-links is recommended to prevent assorted security risks;</span></span><br><span class="line"><span class="comment"># to do so, uncomment this line:</span></span><br><span class="line"><span class="attr">symbolic-links</span> = <span class="number">0</span></span><br><span class="line"></span><br><span class="line"><span class="attr">key_buffer_size</span> = <span class="number">32</span>M</span><br><span class="line"><span class="attr">max_allowed_packet</span> = <span class="number">32</span>M</span><br><span class="line"><span class="attr">thread_stack</span> = <span class="number">256</span>K</span><br><span class="line"><span class="attr">thread_cache_size</span> = <span class="number">64</span></span><br><span class="line"><span class="attr">query_cache_limit</span> = <span class="number">8</span>M</span><br><span class="line"><span class="attr">query_cache_size</span> = <span class="number">64</span>M</span><br><span class="line"><span class="attr">query_cache_type</span> = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"><span class="attr">max_connections</span> = <span class="number">550</span></span><br><span class="line"><span class="comment">#expire_logs_days = 10</span></span><br><span class="line"><span class="comment">#max_binlog_size = 100M</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#log_bin should be on a disk with enough free space.</span></span><br><span class="line"><span class="comment">#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your</span></span><br><span class="line"><span class="comment">#system and chown the specified folder to the mysql user.</span></span><br><span class="line"><span class="attr">log_bin</span>=/var/lib/mysql/mysql_binary_log</span><br><span class="line"></span><br><span class="line"><span class="comment">#In later versions of MySQL, if you enable the binary log and do not set</span></span><br><span class="line"><span class="comment">#a server_id, MySQL will not start. The server_id must be unique within</span></span><br><span class="line"><span class="comment">#the replicating group.</span></span><br><span class="line"><span class="attr">server_id</span>=<span class="number">1</span></span><br><span class="line"></span><br><span class="line"><span class="attr">binlog_format</span> = mixed</span><br><span class="line"></span><br><span class="line"><span class="attr">read_buffer_size</span> = <span class="number">2</span>M</span><br><span class="line"><span class="attr">read_rnd_buffer_size</span> = <span class="number">16</span>M</span><br><span class="line"><span class="attr">sort_buffer_size</span> = <span class="number">8</span>M</span><br><span class="line"><span class="attr">join_buffer_size</span> = <span class="number">8</span>M</span><br><span class="line"></span><br><span class="line"><span class="comment"># InnoDB settings</span></span><br><span class="line"><span class="attr">innodb_file_per_table</span> = <span class="number">1</span></span><br><span class="line"><span class="attr">innodb_flush_log_at_trx_commit</span> = <span class="number">2</span></span><br><span class="line"><span class="attr">innodb_log_buffer_size</span> = <span class="number">64</span>M</span><br><span class="line"><span class="attr">innodb_buffer_pool_size</span> = <span class="number">4</span>G</span><br><span class="line"><span class="attr">innodb_thread_concurrency</span> = <span class="number">8</span></span><br><span class="line"><span class="attr">innodb_flush_method</span> = O_DIRECT</span><br><span class="line"><span class="attr">innodb_log_file_size</span> = <span class="number">512</span>M</span><br><span class="line"></span><br><span class="line"><span class="section">[mysqld_safe]</span></span><br><span class="line"><span class="attr">log-error</span>=/var/log/mysqld.log</span><br><span class="line"><span class="attr">pid-file</span>=/var/run/mysqld/mysqld.pid</span><br><span class="line"></span><br><span class="line"><span class="attr">sql_mode</span>=STRICT_ALL_TABLES</span><br></pre></td></tr></table></figure></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">systemctl restart mysql</span><br></pre></td></tr></table></figure><ol start="8"><li><p>下载jdbc驱动<br>点击<a href="https://dev.mysql.com/downloads/connector/j/5.1.html" target="_blank" rel="noopener">下载</a><br>官方文档推荐版本5.1</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">tar -vxf mysql-connector-java-5.1.47.tar.gz</span><br><span class="line">sudo mkdir -p /usr/share/java/</span><br><span class="line"></span><br><span class="line">sudo cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar</span><br></pre></td></tr></table></figure></li><li><p>为 Cloudera Software 创建数据库<br>Cloudera Manager Server scm scm<br>Activity Monitor amon amon<br>Reports Manager rman rman<br>Hue hue hue<br>Hive Metastore Server metastore hive<br>Sentry Server sentry sentry<br>Cloudera Navigator Audit Server nav nav<br>Cloudera Navigator Metadata Server navms navms<br>Oozie oozie oozie</p></li></ol><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> scm <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> amon <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> rman <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> hue <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> metastore <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> sentry <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> <span class="keyword">nav</span> <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> navms <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">DATABASE</span> oozie <span class="keyword">DEFAULT</span> <span class="built_in">CHARACTER</span> <span class="keyword">SET</span> utf8 <span class="keyword">DEFAULT</span> <span class="keyword">COLLATE</span> utf8_general_ci;</span><br><span class="line"></span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> scm.* <span class="keyword">TO</span> <span class="string">'scm'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'scm'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> amon.* <span class="keyword">TO</span> <span class="string">'amon'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'amon'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> rman.* <span class="keyword">TO</span> <span class="string">'rman'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'rman'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> hue.* <span class="keyword">TO</span> <span class="string">'hue'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'hue'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> metastore.* <span class="keyword">TO</span> <span class="string">'hive'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'hive'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> sentry.* <span class="keyword">TO</span> <span class="string">'sentry'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'sentry'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> nav.* <span class="keyword">TO</span> <span class="string">'nav'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'nav'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> navms.* <span class="keyword">TO</span> <span class="string">'navms'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'navms'</span>;</span><br><span class="line"><span class="keyword">GRANT</span> ALL <span class="keyword">ON</span> oozie.* <span class="keyword">TO</span> <span class="string">'oozie'</span>@<span class="string">'%'</span> <span class="keyword">IDENTIFIED</span> <span class="keyword">BY</span> <span class="string">'oozie'</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 确认是否创建成功</span></span><br><span class="line"><span class="keyword">show</span> <span class="keyword">databases</span>;</span><br><span class="line"><span class="keyword">SHOW</span> <span class="keyword">GRANTS</span> <span class="keyword">FOR</span> <span class="string">'<user>'</span>@<span class="string">'%'</span>;</span><br></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm</span><br></pre></td></tr></table></figure><p><img src="./cdh6.0-install-process/scm-server.png" alt=""></p><ol start="10"><li>安装 cdh 和其他软件<figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sudo systemctl start cloudera-scm-server</span><br><span class="line"><span class="meta">#</span> 查看日志</span><br><span class="line">sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log</span><br></pre></td></tr></table></figure></li></ol><p><img src="./cdh6.0-install-process/start.png" alt=""></p><p>遇到问题：</p><ol><li>Unable to retrieve non-local non-loopback IP address. Seeing address: localhost/127.0.0.1. </li></ol><p><strong>找到问题了，before install 要先设置 network name,我之前没改<br>Set the hostname to a unique name (not localhost).</strong></p><ol start="11"><li>进入web界面<br><img src="./cdh6.0-install-process/webmanager.png" alt=""><br>进入之后先同意协议，选择版本，选择免费或试用。<br>然后提示安装集群，先指定集群的机器。在输入框了里输入几个机器的 Hostname 然后点击搜索，下方会出现列表。<br><img src="./cdh6.0-install-process/searchHost.png" alt=""><br><strong>这里我已经选择过了 所以没勾选</strong></li></ol><p><strong>到一步我建议先去每台集群机器上面安装cloudera-manager-agent</strong><br>登到刚才安装了<code>cloudera-manager-server</code> 的机器，复制 <code>cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm</code> 和<code>cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm</code>两个文件到其他机器<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">scp cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm [email protected]:~</span><br></pre></td></tr></table></figure></p><p>然后在每台机器安装<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">su cdh</span><br><span class="line">cd ~</span><br><span class="line">sudo yum install cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm</span><br><span class="line"></span><br><span class="line">sudo yum install cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm</span><br></pre></td></tr></table></figure></p><p><img src="./cdh6.0-install-process/cdh-and-other-software.png" alt=""><br>下一步选择存储库，直接按推荐的就行，下面几个选项也不用选，因为上面包含了<br>这里 CDH 版本需要配置一下，否则只有 cdh5.14 而且是在线安装<br>首先<a href="https://archive.cloudera.com/cdh6/6.0.0/parcels/" target="_blank" rel="noopener">在这里</a>下载<br><code>CDH-6.0.0-1.cdh6.0.0.p0.537114-el6.parcel</code><br>移动到 <code>/opt/cloudera/parcel-repo</code> 目录下，然后做sha1校验</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 注意文件名</span><br><span class="line">sha1sum CDH-6.0.0-1.cdh6.0.0.p0.339140-el7.parcel | awk '{ print $1 }' > CDH-6.0.0-1.cdh6.0.0.p0.339140-el7.parcel.sha</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 修改文件的用户</span><br><span class="line">sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*</span><br><span class="line"></span><br><span class="line">sudo systemctl restart cloudera-scm-agent.service</span><br></pre></td></tr></table></figure><p>然后刷新一下页面再回到这一步，选择刚才下载的版本<br><img src="./cdh6.0-install-process/indtall-parcel.png" alt=""><br>等待安装。<br><strong>我这里遇到一个情况就是有一个机器没法激活</strong><br>解决办法是卸载 agent 重新安装了<br>如何卸载：<br><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> 检索安装列表</span><br><span class="line">sudo rpm -qa | grep cloudera</span><br><span class="line"><span class="meta">#</span> 卸载agent</span><br><span class="line">sudo rpm -e cloudera-manager-agent-6.0.0-530873.el7.x86_64 --nodeps</span><br><span class="line"><span class="meta">#</span> 删除配置文件</span><br><span class="line">sudo rm -rf /etc/cloudera-scm-agent/config.ini.rpmsave</span><br><span class="line"><span class="meta">#</span> 重新安装</span><br><span class="line">sudo yum install cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm</span><br></pre></td></tr></table></figure></p><p><img src="./cdh6.0-install-process/install-parcel-finished.png" alt=""><br>ok<br>下一步检查机器配置是否正确<br><img src="./cdh6.0-install-process/inspect-host.png" alt=""></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">sysctl vm.swappiness=10</span><br><span class="line">echo never > /sys/kernel/mm/transparent_hugepage/defrag</span><br><span class="line">echo never > /sys/kernel/mm/transparent_hugepage/enabled</span><br><span class="line"><span class="meta">#</span> 添加到开机运行</span><br><span class="line">vim /etc/rc.local </span><br><span class="line">echo never > /sys/kernel/mm/transparent_hugepage/defrag</span><br><span class="line">echo never > /sys/kernel/mm/transparent_hugepage/enabled</span><br></pre></td></tr></table></figure><p>改了以后点击完成就到集群设置页面了<br><img src="./cdh6.0-install-process/select-service.png" alt=""><br>这里有模版也可以全部安装也可以自定义，我就自定义装几个试试<br><img src="./cdh6.0-install-process/host-service.png" alt=""><br>大概这样，这次主要还是 hdfs 所以一共五台机器，可以发现 140 上的服务有点多，所以后面应当多几台机器，不要把dataNode部署到应用服务器上面，因为dataNode会占用IO<br><img src="./cdh6.0-install-process/database-set-up.png" alt=""><br>设置数据库，没啥说的，之前建表的怎么建这里怎么写<br>然后审核更改，可以改一些参数啥的，再下一步开始安装了</p><p><img src="./cdh6.0-install-process/command-detail.png" alt=""><br>都ok，点击继续，完成。</p>]]></content>
<tags>
<tag> hadoop </tag>
</tags>
</entry>
<entry>
<title>使用 Chanell 读取文件到 Buffer 的实例</title>
<link href="/2018/09/20/Java/java-nio/"/>
<url>/2018/09/20/Java/java-nio/</url>
<content type="html"><![CDATA[<h2 id="Channel的实现"><a href="#Channel的实现" class="headerlink" title="Channel的实现"></a>Channel的实现</h2><p> FileChannel 从文件中读写数据。</p><p>DatagramChannel 能通过UDP读写网络中的数据。</p><p>SocketChannel 能通过TCP读写网络中的数据。</p><p>ServerSocketChannel可以监听新进来的TCP连接。</p><h3 id="读取实例"><a href="#读取实例" class="headerlink" title="读取实例"></a>读取实例</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> <span class="keyword">throws</span> IOException </span>{</span><br><span class="line"> RandomAccessFile accessFile = <span class="keyword">new</span> RandomAccessFile(<span class="string">"data/nio-data.txt"</span>,<span class="string">"rw"</span>);</span><br><span class="line"> <span class="comment">// 获取FileChannel</span></span><br><span class="line"> FileChannel inChannel = accessFile.getChannel();</span><br><span class="line"> <span class="comment">// 初始化Buffer</span></span><br><span class="line"> ByteBuffer buffer = ByteBuffer.allocate(<span class="number">48</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 读取数据到Buffer</span></span><br><span class="line"> <span class="keyword">int</span> read = inChannel.read(buffer);</span><br><span class="line"> <span class="keyword">while</span> (read != -<span class="number">1</span>){</span><br><span class="line"> System.out.println(<span class="string">"Read"</span> + read);</span><br><span class="line"> <span class="comment">// 重置buffer的下标，为buffer.hasRemaining()准备</span></span><br><span class="line"> buffer.flip();</span><br><span class="line"></span><br><span class="line"> <span class="keyword">while</span> (buffer.hasRemaining()){</span><br><span class="line"> System.out.println((<span class="keyword">char</span>) buffer.get()); <span class="comment">// 每次读取1字节</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// 清空buffer</span></span><br><span class="line"> buffer.clear();</span><br><span class="line"> <span class="comment">// 继续读取到buffer</span></span><br><span class="line"> read = inChannel.read(buffer);</span><br><span class="line"> }</span><br><span class="line"> accessFile.close();</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure><h3 id="transferFrom复制文件实例"><a href="#transferFrom复制文件实例" class="headerlink" title="transferFrom复制文件实例"></a>transferFrom复制文件实例</h3><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">RandomAccessFile fromFile = <span class="keyword">new</span> RandomAccessFile(<span class="string">"data/fromFile.txt"</span>,<span class="string">"rw"</span>);</span><br><span class="line">FileChannel fromFileChannel = fromFile.getChannel();</span><br><span class="line"></span><br><span class="line">RandomAccessFile toFile = <span class="keyword">new</span> RandomAccessFile(<span class="string">"data/toFile.txt"</span>,<span class="string">"rw"</span>);</span><br><span class="line">FileChannel toFileChannel = toFile.getChannel();</span><br><span class="line"><span class="keyword">long</span> position = <span class="number">0</span>;</span><br><span class="line"><span class="keyword">long</span> count = fromFileChannel.size();</span><br><span class="line"><span class="keyword">long</span> l = toFileChannel.transferFrom(fromFileChannel,position, count );</span><br></pre></td></tr></table></figure>]]></content>
<tags>
<tag> nio </tag>
</tags>
</entry>
<entry>
<title>Hadoop 发行版本对比</title>
<link href="/2018/09/13/Java/Hadoop-version-contrast/"/>
<url>/2018/09/13/Java/Hadoop-version-contrast/</url>
<content type="html"><![CDATA[<h2 id="Cloudera发行版（CDH）"><a href="#Cloudera发行版（CDH）" class="headerlink" title="Cloudera发行版（CDH）"></a>Cloudera发行版（CDH）</h2><p>提供商业服务</p><p>未全部开源</p><p>版本划分：5.x 6.x </p><p>文档：文档分类多 <a href="https://www.cloudera.com/documentation/enterprise/6/6.0.html" target="_blank" rel="noopener">CDH 6 文档</a></p><p>运维：Cloudera Manager</p><p>代码包依赖：编写代码要依赖cdh的版本，否则运行不成功</p><p>二次开发：不支持</p><blockquote><p><strong>Cloudera：</strong>最成型的发行版本，拥有最多的部署案例。提供强大的部署、管理和监控工具。Cloudera开发并贡献了可实时处理大数据的Impala项目。</p></blockquote><h2 id="Hortonworks发行版（HDP）"><a href="#Hortonworks发行版（HDP）" class="headerlink" title="Hortonworks发行版（HDP）"></a>Hortonworks发行版（HDP）</h2><p>提供商业服务</p><p>完全开源 Apache 顶级项目</p><p>版本划分：版本比较新，基本与社区版同步</p><p>文档：文档清晰<a href="https://docs.hortonworks.com/" target="_blank" rel="noopener">Hortonworks文档</a></p><p>运维：Ambari</p><p>代码包依赖：编写代码直接依赖hadoop版本</p><p>二次开发：支持</p><blockquote><p><strong>Hortonworks：</strong>不拥有任何私有（非开源）修改地使用了100%开源Apache Hadoop的唯一提供商。Hortonworks是第一家使用了Apache HCatalog的元数据服务特性的提供商。并且，它们的Stinger开创性地极大地优化了Hive项目。Hortonworks为入门提供了一个非常好的，易于使用的沙盒。Hortonworks开发了很多增强特性并提交至核心主干，这使得Apache Hadoop能够在包括Windows Server和Windows Azure在内的Microsft Windows平台上本地运行。</p></blockquote><p><em>两者都是基于Apache Hadoop社区版衍生出来的</em></p><h2 id="Apache-Hadoop-社区版"><a href="#Apache-Hadoop-社区版" class="headerlink" title="Apache Hadoop 社区版"></a>Apache Hadoop 社区版</h2><p>完全开源免费</p><p>社区活跃</p><p>版本划分：复杂，处理组件版本兼容性</p><p>文档：详细</p><p>部署：复杂，多节点需要分别部署分发配置文件到每一个节点</p><p>运维：对集群的监控，运维，需要安装第三方的其他软件，如ganglia，nagois等，运维难度较大</p><h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><p><a href="https://www.cnblogs.com/kdy11/p/9315706.html" target="_blank" rel="noopener">大数据平台比较-CDH，HDP</a></p>]]></content>
<tags>
<tag> hadoop </tag>
</tags>
</entry>
<entry>
<title>JavaScript 再入门</title>
<link href="/2018/09/06/JavaScript/javaScript/"/>
<url>/2018/09/06/JavaScript/javaScript/</url>
<content type="html"><![CDATA[<p>虽然用项目中使用 jquery 已经写锅很多项目了，但是对于原生 js 很少了解过，趁此机会再了解相关的知识，为之后做铺垫。本文记录平时没接触过，但是可能会在未来开发中有帮助的内容。</p><a id="more"></a><h2 id="javaScript-判断类型"><a href="#javaScript-判断类型" class="headerlink" title="javaScript 判断类型"></a>javaScript 判断类型</h2><ol><li>使用 <code>type of</code></li><li>使用构造函数判断<blockquote><p>平时使用较多的是 <code>type of</code> 这里，这里记录一下如何使用构造函数判断类型</p></blockquote></li></ol><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">foo</span>(<span class="params">param</span>)</span>{</span><br><span class="line"> <span class="keyword">if</span>(param.constructor == <span class="built_in">String</span>){</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">"params is string"</span>)</span><br><span class="line"> }<span class="keyword">else</span> <span class="keyword">if</span>(param.constructor == <span class="built_in">Array</span>){</span><br><span class="line"> <span class="built_in">console</span>.log(<span class="string">"params is array"</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><table><thead><tr><th>param</th><th>Type of</th><th>Constructor</th></tr></thead><tbody><tr><td>{a:’1’}</td><td>object</td><td>Object</td></tr><tr><td>[1,2,3,4]</td><td>object</td><td>Array</td></tr><tr><td>function(){}</td><td>function</td><td>Function</td></tr><tr><td>“string”</td><td>string</td><td>String</td></tr><tr><td>55</td><td>number</td><td>Number</td></tr><tr><td>true</td><td>boolean</td><td>Boolean</td></tr><tr><td>new User()</td><td>object</td><td>User</td></tr></tbody></table><blockquote><p>使用 <code>type of</code> 判断类型时，json对象、数组、自定义对象返回的都是 <code>object</code>；使用构造方法来判断类型可以更加精确</p></blockquote><h2 id="面向对象"><a href="#面向对象" class="headerlink" title="面向对象"></a>面向对象</h2><p>面向对象三大基本特征：封装、继承、多态</p><p>封装：可以通过闭包来实现，</p><p>多态：Javascript 中不像 Java 可以声明多个名称相同参数不同的方法，但是可以在方法内判断参数的个数和类型来实现多态</p><p>继承：实现方式看到了好多种，如下的代码：</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">Person</span>(<span class="params">name</span>) </span>{</span><br><span class="line"> <span class="keyword">this</span>.name = name;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">this</span>.getName = <span class="function"><span class="keyword">function</span> (<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>.name</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">User</span>(<span class="params">name, password</span>) </span>{</span><br><span class="line"> <span class="keyword">this</span>.name = name</span><br><span class="line"> <span class="keyword">this</span>.password = password</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// var u = new User('virle')</span></span><br><span class="line"> <span class="comment">// console.log(u.getName()) // u.getName is not a function</span></span><br><span class="line"></span><br><span class="line"> User.prototype = <span class="keyword">new</span> Person()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> user = <span class="keyword">new</span> User(<span class="string">"virle"</span>)</span><br><span class="line"> <span class="built_in">console</span>.log(user.getName()) <span class="comment">// virle</span></span><br></pre></td></tr></table></figure><blockquote><p>每次创建 <code>User</code> 对象的时候都会创建 <code>Person</code> 对象，包含了 <code>Person</code> 对象中的方法，所以创建的 <code>User</code> 对象可以使用 <code>Person</code> 中的 <code>getName()</code></p></blockquote><p>静态方法声明</p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">User</span>(<span class="params">name,age</span>)</span>{</span><br><span class="line"> <span class="keyword">this</span>.name = name</span><br><span class="line"> <span class="keyword">this</span>.age = age</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">this</span>.getName = <span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>.name</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">this</span>.getAge = <span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">this</span>.age</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">User.cloneUser = <span class="function"><span class="keyword">function</span>(<span class="params">user</span>)</span>{</span><br><span class="line"> <span class="keyword">return</span> <span class="keyword">new</span> User(user.getName(),user.getAge())</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> user = <span class="keyword">new</span> User(<span class="string">'virle'</span>, <span class="number">20</span>)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> clone = User.cloneUser(user)</span><br><span class="line"></span><br><span class="line"><span class="built_in">console</span>.log(clone.getName())</span><br></pre></td></tr></table></figure><blockquote><p>个人感觉 Javascript 面向对象编程写代码很不习惯，用着也很奇怪，可能是习惯了 Java 的一套理念。</p><p>其他参考链接：</p><p><a href="https://www.cnblogs.com/Leo_wl/p/5734794.html" target="_blank" rel="noopener">JS面向对象编程之：封装、继承、多态</a></p><p><a href="https://www.cnblogs.com/z-w-r/p/6739363.html" target="_blank" rel="noopener">基于类（Java）和基于原理（JavaScript）的对象系统的比较</a></p><p><a href="http://es6.ruanyifeng.com/#docs/class" target="_blank" rel="noopener">ECMAScript 6 入门 - Class 的基本语法</a></p><p>ES6 中 Class 的继承比 ES5 的通过修改原型链实现继承，要清晰和方便很多。</p></blockquote><h2 id="命名空间"><a href="#命名空间" class="headerlink" title="命名空间"></a>命名空间</h2><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> VIR = {}</span><br><span class="line"></span><br><span class="line">VIR.util = {}</span><br><span class="line"></span><br><span class="line">VIR.util.Event = {</span><br><span class="line"> addEventListener: <span class="function"><span class="keyword">function</span>(<span class="params"></span>)</span>{...}</span><br><span class="line">}</span><br></pre></td></tr></table></figure><blockquote><p>JavaScript 里并不存在命名空间的概念，但是每个对象都有自己的属性，属性又可以包含对象，所以可以创造出和其他语言类似的命名空间类似的东西</p></blockquote><h2 id="压缩代码"><a href="#压缩代码" class="headerlink" title="压缩代码"></a>压缩代码</h2><ol><li><p>JSMin</p><p>去除所有的非必要的字符，只留下实现功能所必需的代码。JSMin 是通过去掉多余空白字符（包括制表符和换行符）和注释达到压缩的目的</p></li><li><p>Packer</p><p>去除空白和注释，修改变量名，而且最小化代码中的所有单词长度。生成的代码回多处几百字节，用于自解压，对于大文件来说比较合适。</p></li></ol><h2 id="DOM"><a href="#DOM" class="headerlink" title="DOM"></a>DOM</h2><p>DOM 方面的内容接触的比较多，所以只是简单浏览一遍，具体的内容还是推荐<a href="http://www.w3school.com.cn/jsref/dom_obj_document.asp" target="_blank" rel="noopener">DOM参考手册</a></p>]]></content>
<tags>
<tag> JavaScript </tag>
</tags>
</entry>
<entry>
<title>9月书单</title>
<link href="/2018/08/30/book/2018-09/"/>
<url>/2018/08/30/book/2018-09/</url>
<content type="html"><![CDATA[<ol><li>「 写给大家看的设计书（第四版）」</li><li>「 极简主义 」</li><li>「 Picture This, how picture’s work 」<a id="more"></a></li></ol>]]></content>
<tags>
<tag> 阅读 </tag>
</tags>
</entry>
<entry>
<title>Mongodb 复制集部署 + 权限认证</title>
<link href="/2018/08/22/Mongodb/Mongodb-Replset/"/>
<url>/2018/08/22/Mongodb/Mongodb-Replset/</url>
<content type="html"><![CDATA[<p>添加权限认证，创建复制集，Master节点故障自动切换<br><a id="more"></a></p><h2 id="配置环境"><a href="#配置环境" class="headerlink" title="配置环境"></a>配置环境</h2><p>机器：</p><ul><li>host 1:27017 Primary</li><li>host 1:27018 Arbiter</li><li>host 2:27017 Secondary</li></ul><h2 id="目的"><a href="#目的" class="headerlink" title="目的"></a>目的</h2><p>两台机器分别部署两个节点，Primary 提供服务，Secondary 提供备份，当 Primary 宕机时，Secondary 可以代替 Primary 继续提供服务</p><h2 id="步骤"><a href="#步骤" class="headerlink" title="步骤"></a>步骤</h2><ol><li><p>在 host 1 启动 Primary</p></li><li><p>在 host 2 启动 Secondary</p></li><li><p>在 host 1 启动 Arbiter</p></li><li><p>初始化 Replset</p></li><li><p>查看状态</p></li><li><p>测试</p></li></ol><hr><p><strong>本文中 {depPath} 指 mongodb 存放的路径 ，配置文件中为了方便查看，实际使用时请自行补全</strong></p><p>创建 keyfile ，使用 keyfile 校验才能加入复制集</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">pwd </span><br><span class="line"><span class="meta">#</span> {depPath}/mongodb</span><br><span class="line">openssl rand -base64 756 > keyfile</span><br><span class="line">chmod 600 keyfile # 官方文档写的400</span><br></pre></td></tr></table></figure><p>复制keyfile到各实例目录下，注意用户和用户组，保证启动 mongodb 实例的用户有访问权限 keyfile 权限过大无法启动实例</p><p>host1：{depPath}/mongodb/mongodb.cnf</p><p>host2：{depPath}/mongodb/mongodb.cnf</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">port</span>=<span class="number">27017</span></span><br><span class="line"></span><br><span class="line"><span class="attr">dbpath</span>=/mongodb/data/</span><br><span class="line"></span><br><span class="line"><span class="attr">logpath</span>=/mongodb/mongodb.log</span><br><span class="line"></span><br><span class="line"><span class="attr">pidfilepath</span>=/mongodb/mongo.pid</span><br><span class="line"></span><br><span class="line"><span class="attr">logappend</span>=<span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">fork</span>=<span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">bind_ip</span>=<span class="number">0.0</span>.<span class="number">0.0</span></span><br><span class="line"></span><br><span class="line"><span class="comment">#auth=true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">storageEngine</span>=wiredTiger</span><br><span class="line"></span><br><span class="line"><span class="attr">wiredTigerCacheSizeGB</span>=<span class="number">1</span> </span><br><span class="line"></span><br><span class="line"><span class="comment">#keyFile=/mongodb/keyfile</span></span><br><span class="line"></span><br><span class="line"><span class="attr">replSet</span>=reptest</span><br></pre></td></tr></table></figure><p>host1：{depPath}/mongodb_arb/mongodb.cnf</p><figure class="highlight ini"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">port</span>=<span class="number">27017</span></span><br><span class="line"></span><br><span class="line"><span class="attr">dbpath</span>=/mongodb_arb/data/arb</span><br><span class="line"></span><br><span class="line"><span class="attr">logpath</span>=/mongodb_arb/mongodb.log</span><br><span class="line"></span><br><span class="line"><span class="attr">pidfilepath</span>=/mongodb_arb/mongo.pid</span><br><span class="line"></span><br><span class="line"><span class="attr">logappend</span>=<span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">fork</span>=<span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">bind_ip</span>=<span class="number">0.0</span>.<span class="number">0.0</span></span><br><span class="line"></span><br><span class="line"><span class="attr">auth</span>=<span class="literal">true</span></span><br><span class="line"></span><br><span class="line"><span class="attr">storageEngine</span>=wiredTiger</span><br><span class="line"></span><br><span class="line"><span class="attr">wiredTigerCacheSizeGB</span>=<span class="number">1</span> </span><br><span class="line"></span><br><span class="line"><span class="attr">keyFile</span>=/mongodb_arb/keyfile</span><br><span class="line"></span><br><span class="line"><span class="attr">replSet</span>=reptest</span><br></pre></td></tr></table></figure><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> host 1: 启动 Primary , </span><br><span class="line">/mongodb/bin/mongodb -f /mongodb/mongodb.cnf</span><br><span class="line"><span class="meta">#</span> 进入 mongo shell</span><br><span class="line">/mongodb/bin/mongo</span><br><span class="line"><span class="meta">#</span> 首次进入创建用户</span><br><span class="line">use admin</span><br><span class="line"><span class="meta">#</span> 创建 admin 管理员</span><br><span class="line">db.createUser(</span><br><span class="line"> {</span><br><span class="line"> user: "root",</span><br><span class="line"> pwd: "root",</span><br><span class="line"> roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]</span><br><span class="line"> }</span><br><span class="line">)</span><br><span class="line"><span class="meta">#</span> 创建clusterAdmin</span><br><span class="line">db.createUser(</span><br><span class="line">{</span><br><span class="line">user: "dba",</span><br><span class="line">pwd: "dba",</span><br><span class="line">roles: [ { role: "clusterAdmin", db: "admin" } ]</span><br><span class="line">}</span><br><span class="line">)</span><br></pre></td></tr></table></figure><p>首次通过本地的 <code>shell</code>进入，是可以直接创建用户的，之后创建用户需要认证 <code>admin</code> 角色。</p><p>创建完用户把配置文件中的 <code>keyfile</code> 和 <code>auth</code> 注释取消掉，重启 host 1 Primary 实例</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> host 2: 启动 Secondary </span><br><span class="line">/mongodb/bin/mongodb -f /mongodb/mongodb.cnf</span><br><span class="line"><span class="meta">#</span> host 1: 启动 Arbiter</span><br><span class="line">/mongodb_arb/bin/mongodb /mongodb_arb/mongodb.cnf</span><br></pre></td></tr></table></figure><p>在 host 1 进入 mongo shell</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span>## host 和 ip 请自行替换</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 进入 shell</span><br><span class="line">/mongodb/bin/mongo</span><br><span class="line">use admin </span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 认证（clusterAdmin）</span><br><span class="line">db.auth('dba','dba')</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 初始化复制集</span><br><span class="line">rs.initiate(</span><br><span class="line"> {</span><br><span class="line"> _id: "reptest",</span><br><span class="line"> version: 1,</span><br><span class="line"> members: [</span><br><span class="line"> { _id: 0, host : "host 1:27017" },</span><br><span class="line"> { _id: 1, host : "host 2:27017" },</span><br><span class="line"> ]</span><br><span class="line"> }</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 添加一个 Arb 节点</span><br><span class="line">rs.addArb("host 1:27018")</span><br><span class="line"></span><br><span class="line"><span class="meta">#</span> 查看状态</span><br><span class="line">rs.status()</span><br></pre></td></tr></table></figure><p>rs 命令只能在 Primary 节点修改设置，是否为 Primary 显示在命令行头部</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">reptest:PRIMARY></span><br><span class="line">reptest:SECONDARY></span><br></pre></td></tr></table></figure><h2 id="spring-data-mongodb-复制集配置"><a href="#spring-data-mongodb-复制集配置" class="headerlink" title="spring-data-mongodb 复制集配置"></a>spring-data-mongodb 复制集配置</h2><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"><!-- 如果需要验证 使用 credentials 属性 --></span></span><br><span class="line"><span class="tag"><<span class="name">mongo:mongo-client</span> <span class="attr">id</span>=<span class="string">"mongo"</span> <span class="attr">replica-set</span>=<span class="string">"host 1:27017，host 2:27017"</span> <span class="attr">credentials</span>=<span class="string">"databaseName:username@password"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">mongo:client-options</span> <span class="attr">write-concern</span>=<span class="string">"SAFE"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">connections-per-host</span>=<span class="string">"8"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">threads-allowed-to-block-for-connection-multiplier</span>=<span class="string">"4"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">connect-timeout</span>=<span class="string">"1000"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">max-wait-time</span>=<span class="string">"1500"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">socket-keep-alive</span>=<span class="string">"true"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">socket-timeout</span>=<span class="string">"1500"</span></span></span><br><span class="line"><span class="tag"> /></span></span><br><span class="line"><span class="tag"></<span class="name">mongo:mongo-client</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">mongo:db-factory</span> <span class="attr">dbname</span>=<span class="string">"databaseName"</span> <span class="attr">mongo-ref</span>=<span class="string">"mongo"</span> /></span></span><br><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">bean</span> <span class="attr">id</span>=<span class="string">"mongoTemplate"</span> <span class="attr">class</span>=<span class="string">"org.springframework.data.mongodb.core.MongoTemplate"</span>></span></span><br><span class="line"><span class="tag"><<span class="name">constructor-arg</span> <span class="attr">name</span>=<span class="string">"mongoDbFactory"</span> <span class="attr">ref</span>=<span class="string">"mongoDbFactory"</span> /></span></span><br><span class="line"><span class="tag"></<span class="name">bean</span>></span></span><br></pre></td></tr></table></figure><h1 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h1><p><a href="https://docs.mongodb.com/v3.6/tutorial/create-users/#add-users" target="_blank" rel="noopener">Add Users</a></p><p><a href="https://docs.mongodb.com/v3.6/tutorial/enable-authentication/#enable-auth" target="_blank" rel="noopener">Enable Auth</a></p><p><a href="https://docs.mongodb.com/v3.6/tutorial/deploy-replica-set-with-keyfile-access-control/index.html#deploy-new-replica-set-with-keyfile-access-control" target="_blank" rel="noopener">Deploy New Replica Set With Keyfile Access Control</a></p><p><a href="https://docs.mongodb.com/v3.6/tutorial/enforce-keyfile-access-control-in-existing-replica-set/#enforce-keyfile-access-control-in-a-replica-set" target="_blank" rel="noopener">Enforce Keyfile Access Control in a Replica Set</a></p><p><a href="https://docs.mongodb.com/v3.6/replication/#replication" target="_blank" rel="noopener">Replication</a></p><p><a href="https://docs.mongodb.com/v3.6/core/replica-set-arbiter/#replica-set-arbiter" target="_blank" rel="noopener">Replica Set Arbiter</a></p><h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><p>部署时遇到的问题挺多，比如认证是以前就有的，在配置文件中使用了 <code>ReplSet</code> 和 <code>keyfile</code> 后 mongo shell 中执行的命令很多都提示没有认证 <code>admin</code> 权限，后来发现使用 <code>ReplSet</code> 执行命令需要 <code>clusterAdmin</code>。</p><p>本来目的时两个节点可以互相切换。实际操作时一个节点挂掉，另一个并不会自动切换为 PRIMARY ，后来通过文档发现有 <a href="https://docs.mongodb.com/v3.6/core/replica-set-arbiter/#replica-set-arbiter" target="_blank" rel="noopener">Replica Set Arbiter</a> 的概念，因为只有一个节点时无法完成投票选举，需要添加一个 Arbiter 节点来投票。</p><p>多看文档少百度，文章是部署后所写，只提供思路和流程，具体操作请按实际环境进行相应的修改，如有错误之处，还望指出。</p>]]></content>
<tags>
<tag> Mongodb </tag>
</tags>
</entry>
<entry>
<title>Swift URLSession 网络请求</title>
<link href="/2018/08/06/Swift/URLSession/"/>
<url>/2018/08/06/Swift/URLSession/</url>
<content type="html"><![CDATA[<p>使用Swift API 发送请求<br><a id="more"></a></p><h2 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h2><ul><li>macOS High Sierra 10.13.6 </li><li>X Code 9.4.1</li><li>Swift 4.1</li></ul><h2 id="操作"><a href="#操作" class="headerlink" title="操作"></a>操作</h2><ol><li>创建URLSession</li><li>创建URLSessionTask</li><li>捕获Response</li><li>执行Task</li></ol><figure class="highlight swift"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> url = <span class="type">URL</span>(string: <span class="string">"https//www.baidu.com"</span>) <span class="comment">//创建url</span></span><br><span class="line"><span class="comment">// let url = URL(string: "http//www.baidu.com") </span></span><br><span class="line"><span class="comment">//创建dataTask 并捕获结果</span></span><br><span class="line"><span class="keyword">let</span> dataTask = <span class="type">URLSession</span>.shared.dataTask(with: url!) { (data, response, error) <span class="keyword">in</span></span><br><span class="line"><span class="comment">// 错误退出</span></span><br><span class="line"><span class="keyword">if</span> <span class="keyword">let</span> error = error {</span><br><span class="line"><span class="built_in">print</span>(error)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure><figure class="highlight swift"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 状态码判断</span></span><br><span class="line"><span class="keyword">guard</span> <span class="keyword">let</span> httpResponse = response <span class="keyword">as</span>? <span class="type">HTTPURLResponse</span>,(<span class="number">200</span> ... <span class="number">299</span>).<span class="built_in">contains</span>(httpResponse.statusCode) <span class="keyword">else</span> {</span><br><span class="line"><span class="built_in">print</span>(<span class="string">"wrong statusCode"</span>)</span><br><span class="line"><span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="comment">// 判断返回结果类型，输出内容</span></span><br><span class="line"><span class="keyword">if</span> <span class="keyword">let</span> mimeType = httpResponse.mimeType, mimeType == <span class="string">"text/html"</span>,</span><br><span class="line"><span class="keyword">let</span> data = data,</span><br><span class="line"><span class="keyword">let</span> string = <span class="type">String</span>(data: data, encoding: .utf8){</span><br><span class="line"><span class="built_in">print</span>(string)</span><br><span class="line">}</span><br><span class="line">}</span><br><span class="line"><span class="comment">// 执行</span></span><br><span class="line">dataTask.resume()</span><br></pre></td></tr></table></figure><h2 id="可能出现的问题"><a href="#可能出现的问题" class="headerlink" title="可能出现的问题"></a>可能出现的问题</h2><h3 id="http协议的请求无法访问-The-resource-could-not-be-loaded-because-the-App-Transport-Security-policy-requires-the-use-of-a-secure-connection"><a href="#http协议的请求无法访问-The-resource-could-not-be-loaded-because-the-App-Transport-Security-policy-requires-the-use-of-a-secure-connection" class="headerlink" title="http协议的请求无法访问 The resource could not be loaded because the App Transport Security policy requires the use of a secure connection."></a>http协议的请求无法访问 <strong>The resource could not be loaded because the App Transport Security policy requires the use of a secure connection.</strong></h3><p>编辑 <code>info.plist</code> 文件</p><p><code>Information Property List</code> 点击 + ， 添加 <code>App Transport Security Settings</code> ，点击 <code>+</code> 添加子属性 <code>Allow Arbitrary Loads</code> 设置值为 <code>YES</code></p><h3 id="无法连接到网络-找不到-dns-A-server-with-the-specified-hostname-could-not-be-found"><a href="#无法连接到网络-找不到-dns-A-server-with-the-specified-hostname-could-not-be-found" class="headerlink" title="无法连接到网络/找不到 dns A server with the specified hostname could not be found."></a>无法连接到网络/找不到 dns <strong>A server with the specified hostname could not be found.</strong></h3><p>错误日志：</p><blockquote><p>nw_resolver_create_dns_service_locked DNSServiceCreateDelegateConnection failed:</p><p>NSErrorFailingURLStringKey=<a href="http://www.baidu.com/" target="_blank" rel="noopener">http://www.baidu.com/</a>, NSErrorFailingURLKey=<a href="http://www.baidu.com/" target="_blank" rel="noopener">http://www.baidu.com/</a>, _kCFStreamErrorDomainKey=10, _kCFStreamErrorCodeKey=-72000, NSLocalizedDescription=A server with the specified hostname could not be found.} </p></blockquote><p>点击项目左侧导航栏中的项目名称，右侧顶部标签选择 <code>Capabilities</code> 找到第一项 <code>NetWork</code> 勾选 <code>Incoming Connections (Server)</code> 和 <code>Outgoing Connections (Client)</code></p><p>或者干脆直接把App SandBox关了</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>Swift 相关的信息网上比较少，尤其在百度环境下，基本上没找到任何有用的信息，全都是无用的复制粘贴帖子，而且 Swift版本已经更新至 4， 很多方法已经过时或被删除或类名修改。遇到问题最好还是去 Google，Stack Overflow 。</p><p>网上的教程可以作为参考，本文中的代码在以后的版本也许不适用，所以使用还得借助官方的文档，文档可以在官网或 X Code 中找到 （ Shift + Command + 0 / Help - Developer Documentation）</p>]]></content>
<tags>
<tag> Swift </tag>
</tags>
</entry>
<entry>
<title>8月书单</title>
<link href="/2018/07/30/book/2018-08/"/>
<url>/2018/07/30/book/2018-08/</url>
<content type="html"><![CDATA[<ol><li>「 亚瑟王之死（亚瑟王与圆桌骑士的传说） 」</li><li>「 希腊古典神话 」</li><li>「 你所谓的稳定，不过是在浪费生命 」</li><li>「 人类简史 」<a id="more"></a></li></ol>]]></content>
<tags>
<tag> 阅读 </tag>
</tags>
</entry>
<entry>
<title>7月书单</title>
<link href="/2018/06/30/book/2018-07/"/>
<url>/2018/06/30/book/2018-07/</url>
<content type="html"><![CDATA[<ol><li>「 月亮与六便士 」</li><li>「 斜杠青年 」</li><li>「 黑旗：ISIS的崛起 」</li><li>「 人类简史 」</li><li>「 书都不会读，你还想成功 」<a id="more"></a></li></ol>]]></content>
<tags>
<tag> 阅读 </tag>
</tags>
</entry>
<entry>
<title>6月书单</title>
<link href="/2018/05/30/book/2018-06/"/>
<url>/2018/05/30/book/2018-06/</url>
<content type="html"><![CDATA[<ol><li>「 Java多线程编程实战指南 + 设计模式篇 」</li><li>「 三个火枪手 」</li><li>「 麦克白 」</li><li>「 哈姆莱特 」</li><li>「 黑暗时刻：希特勒、大屠杀与纳粹文化 」（没看完）<a id="more"></a></li></ol>]]></content>
<tags>
<tag> 阅读 </tag>
</tags>
</entry>
<entry>
<title>Kafka 机制</title>
<link href="/2018/05/07/Kafka/kafka02/"/>
<url>/2018/05/07/Kafka/kafka02/</url>
<content type="html"><![CDATA[<p>Kafka 运行机制和功能组建的基本了解<br><a id="more"></a></p><h2 id="架构"><a href="#架构" class="headerlink" title="架构"></a>架构</h2><ul><li><p>Broker</p><p>Kafka集群包含一个或多个服务器，这种服务器被称为broker</p></li><li><p>Topic</p><p>发布到集群的消息都有一个类别(物理上不同Topic的消息分开存储，逻辑上一个Topic的消息虽然保存于一个或多个broker上但用户只需指定消息的Topic即可生产或消费数据而不必关心数据存于何处)</p></li><li><p>Partition</p><p>分区每个Topic包含一个或多个Partition</p></li><li><p>Producer</p><p>发布消息到kafka的broker</p></li><li><p>Consumer</p><p>从kafka的broker中获取消息</p></li><li><p>Consumer Group</p><p>每个Consumer属于一个特定的Consumer Group（可为每个Consumer指定group name，若不指定group name则属于默认的group）。</p></li></ul><h2 id="Kafka拓扑结构"><a href="#Kafka拓扑结构" class="headerlink" title="Kafka拓扑结构"></a>Kafka拓扑结构</h2><p>一个典型的Kafka集群中包含若干Producer（可以是web前端产生的Page View，或者是服务器日志，系统CPU、Memory等），若干broker（Kafka支持水平扩展，一般broker数量越多，集群吞吐率越高），若干Consumer Group，以及一个<a href="http://zookeeper.apache.org/" target="_blank" rel="noopener">Zookeeper</a>集群。</p><p>Kafka通过Zookeeper管理集群配置，选举leader，以及在Consumer Group发生变化时进行rebalance。Producer使用push模式将消息发布到broker，Consumer使用pull模式从broker订阅并消费消息。</p><h2 id="Topic-amp-Partition"><a href="#Topic-amp-Partition" class="headerlink" title="Topic & Partition"></a>Topic & Partition</h2><p>Topic在逻辑上可以被认为是一个队列,每条消费都必须指定它的Topic,为了使得Kafka的吞吐率可以线性提高，物理上把Topic分成一个或多个Partition，每个Partition在物理上对应一个文件夹，该文件夹下存储这个Partition的所有消息和索引文件。</p><p>对于传统的message queue而言，一般会删除已经被消费的消息，而Kafka集群会保留所有的消息，无论其被消费与否。当然，因为磁盘限制，不可能永久保留所有数据（实际上也没必要），因此Kafka提供两种策略删除旧数据。一是基于时间，二是基于Partition文件大小。例如可以通过配置$KAFKA_HOME/config/server.properties，让Kafka删除一周前的数据，也可在Partition文件超过1GB时删除旧数据，配置如下所示。</p><h2 id="Consumer-Group"><a href="#Consumer-Group" class="headerlink" title="Consumer Group"></a>Consumer Group</h2><p>使用Consumer high level API时，同一Topic的一条消息只能被同一个Consumer Group内的一个Consumer消费，但多个Consumer Group可同时消费这一消息。</p><h2 id="Push-vs-Pull"><a href="#Push-vs-Pull" class="headerlink" title="Push vs. Pull"></a>Push vs. Pull</h2><p>作为一个消息系统，Kafka遵循了传统的方式，选择由Producer向broker push消息并由Consumer从broker pull消息。</p><p>push模式很难适应消费速率不同的消费者，因为消息发送速率是由broker决定的。push模式的目标是尽可能以最快速度传递消息，但是这样很容易造成Consumer来不及处理消息，典型的表现就是拒绝服务以及网络拥塞。而pull模式则可以根据Consumer的消费能力以适当的速率消费消息。</p><h2 id="Kafka-delivery-guarantee"><a href="#Kafka-delivery-guarantee" class="headerlink" title="Kafka delivery guarantee"></a>Kafka delivery guarantee</h2><ul><li>At most once 消息可能会丢，但绝不会重复传输</li><li>At least one 消息绝不会丢，但可能会重复传输</li><li>Exactly once 每条消息肯定会被传输一次且仅传输一次，很多时候这是用户所想要的。</li></ul>]]></content>
<tags>
<tag> Kafka </tag>
</tags>
</entry>
<entry>
<title>Kafka 集群部署</title>
<link href="/2018/05/03/Kafka/kafka01/"/>
<url>/2018/05/03/Kafka/kafka01/</url>
<content type="html"><![CDATA[<p>Kafka 集群环境的搭建过程<br><a id="more"></a></p><h2 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h2><p>Apache Kafka® is a distributed streaming platform. 是一个分布式流平台</p><p>A streaming platform has three key capabilities: 流媒体平台具有三个关键功能</p><ul><li>Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system. 发布和订阅记录流，类似于消息队列或企业消息系统。</li><li>Store streams of records in a fault-tolerant durable way.以容错持久的方式存储记录的流。</li><li>Process streams of records as they occur.处理记录发生时的流</li></ul><h2 id="使用场景"><a href="#使用场景" class="headerlink" title="使用场景"></a>使用场景</h2><ul><li><p>消息队列(MQ)</p></li><li><p>分布式日志系统(Log)</p></li><li><p>数据通道(Messaging)</p></li></ul><p>参考:</p><p><a href="https://yq.aliyun.com/articles/60229" target="_blank" rel="noopener">KafKa的介绍以及使用场景</a></p><p><a href="https://blog.csdn.net/mr_hou2016/article/details/79653242" target="_blank" rel="noopener">日志收集为什么用kafka</a></p><h2 id="依赖"><a href="#依赖" class="headerlink" title="依赖"></a>依赖</h2><ul><li><p>java(kafka使用scale语言可以运行在jvm)</p></li><li><p>zookeeper(高性能分布式应用协调服务)</p><blockquote><p>早期版本的kafka用zookeeper做meta信息存储，consumer的消费状态，group的管理以及 offset的值。考虑到zookeeper本身的一些因素以及整个架构较大概率存在单点问题，新版本中确实逐渐弱化了zookeeper的作用</p></blockquote></li></ul><h2 id="元素"><a href="#元素" class="headerlink" title="元素"></a>元素</h2><ul><li><p>broker:对于KafKa集群来说,每一个KafKa实例都被称为一个broker</p></li><li><p>消费者(Consumer)</p></li><li><p>生产者(Producer)</p></li><li><p>话题（Topic)</p><p>相当于一个类别,topic之间是隔绝的,生产者可以push到某个topic,消费者可以从某个topic获取</p></li><li><p>分区(Partition)</p><p>一个类别下的分区,生产者可以向指定的分区存.多个分区可以增加吞吐量</p></li></ul><h2 id="配置"><a href="#配置" class="headerlink" title="配置"></a>配置</h2><h3 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h3><p>机器一:</p><ul><li><p>ip: 10.37.129.3</p></li><li><p>zookeeper</p></li></ul><p>机器二: </p><ul><li><p>ip : 10.37.129.4</p></li><li><p>kafka id=0</p></li></ul><p>机器三:</p><ul><li><p>ip: 10.37.129.5</p></li><li><p>kafka id=1</p></li></ul><p>三台机器都需要配置java环境</p><h4 id="zookeeper配置"><a href="#zookeeper配置" class="headerlink" title="zookeeper配置"></a>zookeeper配置</h4><ul><li><p>配置java环境</p></li><li><p><a href="http://zookeeper.apache.org/releases.html#download" target="_blank" rel="noopener">下载zookeeper</a></p></li><li><p>安装</p><p><code>tar -zxf zookeeper-3.4.12.tar.gz</code></p><p><code>cd zookeeper-3.4.12</code></p></li><li><p>修改配置</p><p><code>vim conf/zoo.cfg</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span> The number of milliseconds of each tick </span><br><span class="line">tickTime=2000 </span><br><span class="line"><span class="meta">#</span> The number of ticks that the initial </span><br><span class="line"><span class="meta">#</span> synchronization phase can take </span><br><span class="line">initLimit=10 </span><br><span class="line"><span class="meta">#</span> The number of ticks that can pass between </span><br><span class="line"><span class="meta">#</span> sending a request and getting an acknowledgement </span><br><span class="line">syncLimit=5 </span><br><span class="line"><span class="meta">#</span> the directory where the snapshot is stored. </span><br><span class="line"><span class="meta">#</span> do not use /tmp for storage, /tmp here is just </span><br><span class="line"><span class="meta">#</span> example sakes. </span><br><span class="line">dataDir=/home/virle/data/zookeeper </span><br><span class="line"><span class="meta">#</span> the port at which the clients will connect </span><br><span class="line">clientPort=2181</span><br></pre></td></tr></table></figure><p>保存配置文件</p></li><li><p>启动</p><p><code>bin/zkServer.sh start</code></p></li><li><p>查看状态 </p><p><code>bin/zkServer.sh statu</code></p><blockquote><p>ZooKeeper JMX enabled by default</p><p>Using config: /home/virle/zookeeper-3.4.12/bin/../conf/zoo.cfg</p><p>Mode: standalone</p></blockquote><p>显示 standalone 启动成功</p></li></ul><h4 id="kafka-配置"><a href="#kafka-配置" class="headerlink" title="kafka 配置"></a>kafka 配置</h4><ul><li><p>配置java环境</p></li><li><p><a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz" target="_blank" rel="noopener">下载</a></p></li><li><p>安装</p><p><code>tar</code> <code>-xzf kafka_2.11-1.1.0.tgz</code></p><p><code>cd kafka_2.11-1.1.0</code></p></li><li><p>配置文件</p><p><code>vim config/server.properties</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">broker.id=0 # 集群中唯一标识</span><br><span class="line">num.partitions=1 # 可以设置分区数量</span><br><span class="line">log.dirs=/home/virle/logs/kafka1-logs # 日志路径 默认在/tmp</span><br><span class="line">zookeeper.connect=10.37.129.3:2181 # zookeeper服务器</span><br><span class="line"><span class="meta">#</span> 去掉这两项的#将主机名修改为服务器的ip地址</span><br><span class="line">listeners=PLAINTEXT://10.37.129.4:9092</span><br><span class="line">advertised.listeners=PLAINTEXT://10.37.129.4:9092</span><br></pre></td></tr></table></figure></li><li><p>启动</p><p><code>bin/kafka-server-start.sh config/server.properties</code></p><blockquote><p>[2018-05-04 04:30:16,300] INFO Kafka version : 1.1.0 (org.apache.kafka.common.utils.AppInfoParser)</p><p>[2018-05-04 04:30:16,302] INFO Kafka commitId : fdcf75ea326b8e07 (org.apache.kafka.common.utils.AppInfoParser)</p><p>[2018-05-04 04:30:16,323] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)</p></blockquote><p>显示 [KafkaServer id=0] started 则启动成功</p></li></ul><p><strong>两台服务器使用相同的方式进行配置,修改地方是broker.id和listener中的ip地址</strong></p><h2 id="kafka-命令行测试"><a href="#kafka-命令行测试" class="headerlink" title="kafka 命令行测试"></a>kafka 命令行测试</h2><p>启动消费者</p><p><code>bin/kafka-console-consumer.sh --bootstrap-server 10.37.129.4:9092 --topic test --from-beginning</code></p><p>在新的shell中启动生产者</p><p><code>bin/kafka-console-producer.sh --broker-list 10.37.129.4:9092 --topic test</code></p><p>输入一个消息并回车</p><p>此时应该可以在消费者的命令行看到消息</p><h2 id="Java-客户端-连接测试"><a href="#Java-客户端-连接测试" class="headerlink" title="Java 客户端 连接测试"></a>Java 客户端 连接测试</h2><ul><li><p>引入依赖</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.kafka<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>kafka-clients<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.1.0<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"><span class="tag"></<span class="name">dependency</span>></span></span><br></pre></td></tr></table></figure></li><li><p>java client</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line">props.put(<span class="string">"bootstrap.servers"</span>, <span class="string">"10.37.129.4:9092"</span>);</span><br><span class="line">props.put(<span class="string">"transactional.id"</span>, <span class="string">"this"</span>);</span><br><span class="line">AdminClient adminClient = AdminClient.create(props);</span><br><span class="line">ArrayList<NewTopic> topics = <span class="keyword">new</span> ArrayList<NewTopic>();</span><br><span class="line"><span class="comment">//创建新的topic</span></span><br><span class="line">NewTopic newTopic = <span class="keyword">new</span> NewTopic(<span class="string">"test01"</span>,<span class="number">1</span>,(<span class="keyword">short</span>)<span class="number">1</span>);</span><br><span class="line">topics.add(newTopic);</span><br><span class="line">adminClient.createTopics(topics);</span><br><span class="line"><span class="comment">//删除topic</span></span><br><span class="line">adminClient.deleteTopics(Arrays.asList(<span class="string">"test01"</span>));</span><br><span class="line"><span class="comment">//遍历topic</span></span><br><span class="line">ListTopicsResult listTopicsResult = adminClient.listTopics();</span><br><span class="line">KafkaFuture<Set<String>> names = listTopicsResult.names();</span><br><span class="line">Set<String> strings = names.get();</span><br><span class="line">strings.forEach(str-> System.out.println(str));</span><br></pre></td></tr></table></figure></li><li><p>生产者</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line">props.put(<span class="string">"bootstrap.servers"</span>, <span class="string">"10.37.129.4:9092"</span>); <span class="comment">//修改为服务器的ip 多个服务器可以使用","隔开</span></span><br><span class="line">props.put(<span class="string">"acks"</span>, <span class="string">"all"</span>);</span><br><span class="line">props.put(<span class="string">"retries"</span>, <span class="number">0</span>);</span><br><span class="line">props.put(<span class="string">"batch.size"</span>, <span class="number">16384</span>);</span><br><span class="line">props.put(<span class="string">"linger.ms"</span>, <span class="number">1</span>);</span><br><span class="line">props.put(<span class="string">"buffer.memory"</span>, <span class="number">33554432</span>);</span><br><span class="line">props.put(<span class="string">"key.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line">props.put(<span class="string">"value.serializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringSerializer"</span>);</span><br><span class="line"></span><br><span class="line">Producer<String, String> producer = <span class="keyword">new</span> KafkaProducer<String, String>(props);</span><br><span class="line"><span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i < <span class="number">10</span>; i++) {</span><br><span class="line"> producer.send(<span class="keyword">new</span> ProducerRecord<String, String>(<span class="string">"test"</span>, Integer.toString(i), Integer.toString(i)));</span><br><span class="line">}</span><br><span class="line">producer.close();</span><br></pre></td></tr></table></figure></li><li><p>消费者</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line">Properties props = <span class="keyword">new</span> Properties();</span><br><span class="line">props.put(<span class="string">"bootstrap.servers"</span>, <span class="string">"10.37.129.4:9092"</span>); <span class="comment">//修改为服务器ip 多个可用","隔开</span></span><br><span class="line">props.put(<span class="string">"group.id"</span>, <span class="string">"test"</span>);</span><br><span class="line">props.put(<span class="string">"enable.auto.commit"</span>, <span class="string">"true"</span>);</span><br><span class="line">props.put(<span class="string">"auto.commit.interval.ms"</span>, <span class="string">"1000"</span>);</span><br><span class="line">props.put(<span class="string">"key.deserializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringDeserializer"</span>);</span><br><span class="line">props.put(<span class="string">"value.deserializer"</span>, <span class="string">"org.apache.kafka.common.serialization.StringDeserializer"</span>);</span><br><span class="line">KafkaConsumer<String, String> consumer = <span class="keyword">new</span> KafkaConsumer<>(props);</span><br><span class="line">consumer.subscribe(Arrays.asList(<span class="string">"test"</span>));</span><br><span class="line"><span class="keyword">while</span> (<span class="keyword">true</span>) {</span><br><span class="line"> ConsumerRecords<String, String> records = consumer.poll(<span class="number">100</span>);</span><br><span class="line"> <span class="keyword">for</span> (ConsumerRecord<String, String> record : records){</span><br><span class="line"> System.out.printf(<span class="string">"offset = %d, key = %s, value = %s%n"</span>, record.offset(), record.key(), record.value());</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></li></ul><p>启动消费者程序进入等待,启动生产者,可以看到消费者控制台输出</p><h1 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h1><p><a href="https://blog.csdn.net/lizhitao/article/details/23744675" target="_blank" rel="noopener">kafka在zookeeper中存储结构</a></p><p><a href="https://blog.csdn.net/unionibm/article/details/54090235" target="_blank" rel="noopener">为什么Kafka可以高效快速的存储</a></p><p><a href="https://blog.csdn.net/qinzhaokun/article/details/50486916" target="_blank" rel="noopener">Kafka原理及应用</a></p>]]></content>
<tags>
<tag> Kafka </tag>
</tags>
</entry>
<entry>
<title>5月书单</title>
<link href="/2018/04/30/book/2018-05/"/>
<url>/2018/04/30/book/2018-05/</url>
<content type="html"><![CDATA[<ol><li>「 三体全集 」（还差最后一本）</li><li>「 Think in JavaScript 」（没看完）</li><li>「 代码简洁之道 」</li><li>「 重口味心理学2 」<a id="more"></a></li></ol>]]></content>
<tags>
<tag> 阅读 </tag>
</tags>
</entry>
<entry>
<title>R 语言连接数据库</title>
<link href="/2018/04/12/R/R-conenct-to-databaseI/"/>
<url>/2018/04/12/R/R-conenct-to-databaseI/</url>
<content type="html"><![CDATA[<p>最近研究R连接数据库的方法，网上找到的帖子都千篇一律，经过测试整合，整理一下过程<br><a id="more"></a></p><h2 id="连接mysql"><a href="#连接mysql" class="headerlink" title="连接mysql"></a>连接mysql</h2><p>mysql的连接比较简单，导入RMySQL的包直接就可以连接</p><figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">installl.packages(<span class="string">"RMySQL"</span>)</span><br><span class="line"><span class="keyword">library</span>(<span class="string">"RMySQL"</span>)</span><br><span class="line">conn = dbConnect(MySQL(),dbname=<span class="string">"r_test"</span>,password=<span class="string">"123456"</span>,host=<span class="string">"localhost"</span>,username=<span class="string">"root"</span>)</span><br></pre></td></tr></table></figure><h3 id="获取数据的方式"><a href="#获取数据的方式" class="headerlink" title="获取数据的方式"></a>获取数据的方式</h3><ul><li><p>dbGetQuery(conn,statement)</p><figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mydate <- dbGetQuery(conn,<span class="string">"select * from test_table"</span>)</span><br></pre></td></tr></table></figure></li><li><p>dbSendQuery(conn,statement)</p> <figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">res <- dbSendQuery(conn,<span class="string">"select * from test_table"</span>)</span><br><span class="line">mydata <- dbFetch(res,n=-<span class="number">1</span>)</span><br></pre></td></tr></table></figure><p> n是获取几条记录，-1为剩余所有</p></li></ul><h2 id="连接oracle"><a href="#连接oracle" class="headerlink" title="连接oracle"></a>连接oracle</h2><h3 id="首先要配置odbc数据源"><a href="#首先要配置odbc数据源" class="headerlink" title="首先要配置odbc数据源"></a>首先要配置odbc数据源</h3><ol><li>打开Oracle配置集成工具中的ODBC<strong>数据源管理器</strong></li><li>添加新的配置，选择oraclexxxx</li><li>DataSourceName 填Oracle （此处的值是数据源的值，在创建连接时需要使用）</li><li>Description 随意填写</li><li>TNS Service Name: 其实就是 ip / 实例名 （例如 172.17.17.1/erma）</li><li>UserID 连接oracle的用户名</li><li>点击 TestConnection 测试连接是否可用</li><li>输入密码 点击ok</li><li>显示connetcion successful</li></ol><p>此时已经可以在R中连接了</p><p>在终端输入 R</p><figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">library</span>(ODBC)</span><br><span class="line"><span class="comment">#此处第一个参数“Oracle” 对应上面创建时填写的数据源</span></span><br><span class="line">myconn <- odbcConnect(<span class="string">"Oracle"</span>,uid=<span class="string">""</span>,pwd=<span class="string">""</span>,believeNRows = <span class="literal">FALSE</span>)</span><br><span class="line">sqlQuery(conn,<span class="string">"select * from emp"</span>)</span><br></pre></td></tr></table></figure><h1 id="Attention"><a href="#Attention" class="headerlink" title="Attention"></a>Attention</h1><ul><li><p>此处注意 如果不加believeNRows = FALSE 可能会查不到数据</p><blockquote><p><a href="https://stackoverflow.com/questions/7425100/rodbc-queries-returning-zero-rows" target="_blank" rel="noopener">stackoverflow:Issue: RODBC (falsely) returning zero rows</a></p><p>R连接oracle数据库Error in odbcFetchRows negative length vectors are not allowed</p></blockquote></li><li><p>连接oracle时要开启服务，并且保证端口可用（排除防火墙）实例名要正确 </p></li></ul><p>连接成功后可以进行一些操作，如获取表数据之类，由于我只使用了sql查询，所以对其他方法没有细致的研究，有需要可以参考文档</p><p><a href="https://www.rdocumentation.org/packages/RMySQL/versions/0.10.13#functions" target="_blank" rel="noopener">Functions in RMySQL</a></p><p><a href="https://www.rdocumentation.org/packages/RODBC/versions/1.3-15#functions" target="_blank" rel="noopener">Functions in RODBC</a></p><p>以上</p>]]></content>
<tags>
<tag> R </tag>
</tags>
</entry>
<entry>
<title>数据分析中对缺失数据的处理方式</title>
<link href="/2018/04/12/R/R-deilmethods/"/>
<url>/2018/04/12/R/R-deilmethods/</url>
<content type="html"><![CDATA[<p>数据处理前对缺失数据的几种处理方式<br><a id="more"></a></p><ul><li><p>当缺失数据较少时直接删除相应样本</p><p> 删除缺失数据样本，其前提是缺失数据的比例较少，而且缺失数据是随机出现的，这样删除缺失数据后对分析结果影响不大。如果缺失数据大于30%应当删除该特征</p></li><li><p>用上下数据进行填充</p><p> data.fillna(method=’pad’) or (method=’bfill’)</p></li><li><p>对缺失数据进行插补</p><p> 用均值、中位数、众数、随机值等统计值填充，其优点在于不会减少样本信息，处理简单。但是缺点在于当缺失数据不是随机出现时会产成偏误。</p><p> 多重插补法（Multiple imputation）：多重插补是通过变量间关系来预测缺失数据，利用蒙特卡罗方法生成多个完整数据集，再对这些数据集分别进行分析，最后对这些分析结果进行汇总处理。可以用mice包实现。</p></li><li><p>使用对缺失数据不敏感的分析方法，例如决策树</p></li></ul>]]></content>
<tags>
<tag> R </tag>
</tags>
</entry>
<entry>
<title>如何在 Java 项目中调用 R</title>
<link href="/2018/04/12/Java/how-to-invoking-R-method-in-java/"/>
<url>/2018/04/12/Java/how-to-invoking-R-method-in-java/</url>
<content type="html"><![CDATA[<p>Web 项目中需要调用 R 来进行数据分析，记录最基本的调用过程<br><a id="more"></a></p><p>环境：macos 10.12.5</p><h2 id="安装R环境"><a href="#安装R环境" class="headerlink" title="安装R环境"></a>安装R环境</h2><p><a href="https://cran.r-project.org/bin/macosx/" target="_blank" rel="noopener">Download</a><br>同意协议并安装<br>打开终端 </p><h2 id="安装Rserver包"><a href="#安装Rserver包" class="headerlink" title="安装Rserver包"></a>安装Rserver包</h2><p><code>cd /Applications/R.app/Contents/MacOS</code> 进入R应用目录<br><code>R CMD</code> 启动R命令行模式<br>下载<code>install.packages("Rserve")</code> </p><h2 id="启动Rserver服务"><a href="#启动Rserver服务" class="headerlink" title="启动Rserver服务"></a>启动Rserver服务</h2><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">library("Rserve")</span><br><span class="line"></span><br><span class="line">Rserve()</span><br></pre></td></tr></table></figure><p>显示</p><blockquote><p>Rserv started in daemon mode. </p></blockquote><p>即表示启动成功 </p><h2 id="创建java项目"><a href="#创建java项目" class="headerlink" title="创建java项目"></a>创建java项目</h2><ul><li><p>maven配置</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><?xml version="1.0" encoding="UTF-8"?></span><br><span class="line"><span class="tag"><<span class="name">project</span> <span class="attr">xmlns</span>=<span class="string">"http://maven.apache.org/POM/4.0.0"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xsi:schemaLocation</span>=<span class="string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">rjava.version</span>></span>0.9-7<span class="tag"></<span class="name">rjava.version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"> <span class="comment"><!-- rjava --></span></span><br><span class="line"> <span class="comment"><!-- https://mvnrepository.com/artifact/com.github.lucarosellini.rJava/JRIEngine --></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.github.lucarosellini.rJava<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>JRIEngine<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${rjava.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.github.lucarosellini.rJava<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>REngine<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${rjava.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.github.lucarosellini.rJava<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>JRI<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${rjava.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>com.github.yannrichet<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>Rsession<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.8.3<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>junit<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>junit<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>4.12<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependencies</span>></span></span><br><span class="line"><span class="tag"></<span class="name">project</span>></span></span><br></pre></td></tr></table></figure></li><li><p>创建测试类<br>TestR.java</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.junit.Test;</span><br><span class="line"><span class="keyword">import</span> org.rosuda.REngine.REXP;</span><br><span class="line"><span class="keyword">import</span> org.rosuda.REngine.REXPMismatchException;</span><br><span class="line"><span class="keyword">import</span> org.rosuda.REngine.Rserve.RConnection;</span><br><span class="line"><span class="keyword">import</span> org.rosuda.REngine.Rserve.RserveException;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">Test1</span> </span>{</span><br><span class="line"> <span class="meta">@Test</span></span><br><span class="line"> <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testConnect</span><span class="params">()</span></span>{</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> RConnection c = <span class="keyword">new</span> RConnection();</span><br><span class="line"> REXP x = c.eval(<span class="string">"R.version.string"</span>);</span><br><span class="line"> System.out.println(x.asString());</span><br><span class="line"> } <span class="keyword">catch</span> (RserveException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> } <span class="keyword">catch</span> (REXPMismatchException e) {</span><br><span class="line"> e.printStackTrace();</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure><p>运行测试<br>显示 R version 3.4.1<br>执行语句 eval<br>session.asString(“statement”)</p></li></ul>]]></content>
<tags>
<tag> R </tag>
</tags>
</entry>
<entry>
<title>R 语言服务器环境部署</title>
<link href="/2018/04/12/R/R-environment-setup/"/>
<url>/2018/04/12/R/R-environment-setup/</url>
<content type="html"><![CDATA[<p>配置环境变量</p><p>R_HOME = R语言的安装路径</p><p>path 在末尾追加“;”号 %R_HOME%/bin</p><h1 id="安装所需的包"><a href="#安装所需的包" class="headerlink" title="安装所需的包"></a>安装所需的包</h1><p>其中用来连接的包叫Rserve</p><p>命令行输入R进入</p><p>输入</p><figure class="highlight r"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">install.packages(<span class="string">'Rserve'</span>) <span class="comment">#java同个Rserve服务连接到R</span></span><br><span class="line"></span><br><span class="line">install.packages(<span class="string">'RMySQL'</span>) <span class="comment">#R语言连接mysql</span></span><br><span class="line"></span><br><span class="line">install.packages(<span class="string">'rjson'</span>) <span class="comment">#将数据转换为json格式</span></span><br><span class="line"></span><br><span class="line">install.packages(<span class="string">"forecast"</span>) <span class="comment">#R语言模型预测的包</span></span><br></pre></td></tr></table></figure><h1 id="启动远程访问服务"><a href="#启动远程访问服务" class="headerlink" title="启动远程访问服务"></a>启动远程访问服务</h1><p>进入R的library目录</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd library/Rserve/libs/x64</span><br><span class="line">R CMD Rserve --RS-enable-remote #开启远程访问</span><br></pre></td></tr></table></figure><h1 id="Attention"><a href="#Attention" class="headerlink" title="Attention"></a>Attention</h1><ul><li>在R中加载类并使用Rserve()方法启动以及在Rserve目录使用R CMD Rserve 均为本地访问，远程访问需要加后面的参数</li></ul>]]></content>
<tags>
<tag> R </tag>
</tags>
</entry>
<entry>
<title>Forecast结果的术语</title>
<link href="/2017/08/08/R/R-words-note/"/>
<url>/2017/08/08/R/R-words-note/</url>
<content type="html"><![CDATA[<p>forecast(x,level=置信度,h=周期)<br>accuracy()<br>ME：平均误差<br>RMSE：均方根误差<br>MAE：平均绝对误差<br>MPE：平均百分比误差<br>MAPE：平均绝对误差百分比<br>MASE：绝对尺度误差<br>ACF1：在滞后1误差自相关。 </p><p>fore$level 置信度<br>fore$lower 预测下界<br>fore$upper 预测上界<br>fore$mean 预测值</p><p>#回归分析</p><p>回归分析研究的主要步骤：</p><ol><li>确定因变量Y 与 自变量X1, X2, …, Xn 之间的定量关系表达式，即回归方程。</li><li>对回归方程的置信度检查。</li><li>判断自变量Xn(n=1,2,…,m)对因变量的影响。</li><li>利用回归方程进行预测。</li></ol>]]></content>
<tags>
<tag> R </tag>
</tags>
</entry>
</search>