-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
211 lines (177 loc) · 15.3 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="description" content="None" />
<link rel="shortcut icon" href="img/favicon.ico" />
<title>Keras batchflow</title>
<link rel="stylesheet" href="css/theme.css" />
<link rel="stylesheet" href="css/theme_extra.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.8.0/styles/github.min.css" />
<link href="css/custom.css" rel="stylesheet" />
<script>
// Current page data
var mkdocs_page_name = "Home";
var mkdocs_page_input_path = "index.md";
var mkdocs_page_url = null;
</script>
<!--[if lt IE 9]>
<script src="js/html5shiv.min.js"></script>
<![endif]-->
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.8.0/highlight.min.js"></script>
<script>hljs.highlightAll();</script>
</head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="." class="icon icon-home"> Keras batchflow
</a>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal current" href="#">Home</a>
<ul class="current">
</ul>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="user_guide/">User guide</a>
</li>
</ul>
<p class="caption"><span class="caption-text">Examples</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="titanic_example/">Basic use of Keras-batchflow with Titanic data</a>
</li>
</ul>
<p class="caption"><span class="caption-text">API</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="batch_generators_api/">Batch generators</a>
</li>
<li class="toctree-l1"><a class="reference internal" href="batch_transformers_api/">Batch transformers</a>
</li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" role="navigation" aria-label="Mobile navigation menu">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href=".">Keras batchflow</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content"><div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="." class="icon icon-home" aria-label="Docs"></a></li>
<li class="breadcrumb-item active">Home</li>
<li class="wy-breadcrumbs-aside">
<a href="https://github.com/maxsch3/keras-batchflow/edit/master/docs/index.md" class="icon icon-github"> Edit on GitHub</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div class="section" itemprop="articleBody">
<h1 id="overview">Overview</h1>
<p>Keras batchflow is a batch generator framework for <a href="https://keras.io">Keras</a>.
The framework is generating batches for keras fit_generator and predict_generator functions in all sorts of less
standard scenarios: multi-input and multi-output scenarios, scenarios employing dynamic data augmentation with dependencies
between variables, etc.</p>
<p>The framework bridges gaps between keras and other two core data science modules: pandas and sklearn. With it, you can
use pandas dataframe directly as a datasource for your keras model. You can use all breadth of standard sklearn encoders
to transform columns of a dataframe into numpy arrays. </p>
<p>Following keras's modular approach, keras batchflow is built in the same way. A batch generator the framework produces
is actually
a stack of few interchangeable objects selected from a library, just like keras's layers. Each selected object adds new
feature to the generator. All of this gives combinatorial strength to the framework which is capable covering a wide range of
scenarios by just mixing library objects. </p>
<h1 id="quick-taster-example">Quick taster example</h1>
<p>Here is a quick example of what the framework can do:</p>
<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">LabelEncoder</span><span class="p">,</span> <span class="n">LabelBinarizer</span>
<span class="kn">from</span> <span class="nn">keras_batchflow.batch_generators</span> <span class="kn">import</span> <span class="n">BatchGenerator</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span>
<span class="s1">'var1'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'Class 0'</span><span class="p">,</span> <span class="s1">'Class 1'</span><span class="p">,</span> <span class="s1">'Class 0'</span><span class="p">,</span> <span class="s1">'Class 2'</span><span class="p">,</span> <span class="s1">'Class 0'</span><span class="p">,</span> <span class="s1">'Class 1'</span><span class="p">,</span> <span class="s1">'Class 0'</span><span class="p">,</span> <span class="s1">'Class 2'</span><span class="p">],</span>
<span class="s1">'var2'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'Green'</span><span class="p">,</span> <span class="s1">'Yellow'</span><span class="p">,</span> <span class="s1">'Red'</span><span class="p">,</span> <span class="s1">'Brown'</span><span class="p">,</span> <span class="s1">'Green'</span><span class="p">,</span> <span class="s1">'Yellow'</span><span class="p">,</span> <span class="s1">'Red'</span><span class="p">,</span> <span class="s1">'Brown'</span><span class="p">],</span>
<span class="s1">'label'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'Leaf'</span><span class="p">,</span> <span class="s1">'Flower'</span><span class="p">,</span> <span class="s1">'Leaf'</span><span class="p">,</span> <span class="s1">'Branch'</span><span class="p">,</span> <span class="s1">'Green'</span><span class="p">,</span> <span class="s1">'Yellow'</span><span class="p">,</span> <span class="s1">'Red'</span><span class="p">,</span> <span class="s1">'Brown'</span><span class="p">]</span>
<span class="p">})</span>
<span class="c1">#prefit sklearn encoders</span>
<span class="n">var1_enc</span> <span class="o">=</span> <span class="n">LabelEncoder</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">[</span><span class="s1">'var1'</span><span class="p">])</span>
<span class="n">var2_enc</span> <span class="o">=</span> <span class="n">LabelEncoder</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">[</span><span class="s1">'var2'</span><span class="p">])</span>
<span class="n">label_enc</span> <span class="o">=</span> <span class="n">LabelBinarizer</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">df</span><span class="p">[</span><span class="s1">'label'</span><span class="p">])</span>
<span class="c1"># define a batch generator</span>
<span class="n">train_gen</span> <span class="o">=</span> <span class="n">BatchGenerator</span><span class="p">(</span>
<span class="n">df</span><span class="p">,</span>
<span class="n">x_structure</span><span class="o">=</span><span class="p">((</span><span class="s1">'var1'</span><span class="p">,</span> <span class="n">var1_enc</span><span class="p">),</span> <span class="p">(</span><span class="s1">'var2'</span><span class="p">,</span> <span class="n">var2_enc</span><span class="p">)),</span>
<span class="n">y_structure</span><span class="o">=</span><span class="p">(</span><span class="s1">'label'</span><span class="p">,</span> <span class="n">label_enc</span><span class="p">),</span>
<span class="n">batch_size</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span>
<span class="n">train_mode</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
</code></pre></div>
<p>The generator returns batches of format (x_structure, y_structure) and the shape of the batches is:</p>
<div class="codehilite"><pre><span></span><code><span class="o">>>></span> <span class="n">train_gen</span><span class="o">.</span><span class="n">shape</span>
<span class="p">(((</span><span class="kc">None</span><span class="p">,</span> <span class="p">),</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="p">)),</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
</code></pre></div>
<p>The first element is a x_structure and it is a list if two inputs. Both of them are outputs of LabelEncoders, that
return integer ids of categorical variables, hence only one dimension. The y_structure is a single output produced by
one-hot encoder, hence 2 dimensions.</p>
<p>Now you can define a neural network and use above generator in <code>fit_generator</code> to train it:</p>
<div class="codehilite"><pre><span></span><code><span class="kn">from</span> <span class="nn">keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span><span class="p">,</span> <span class="n">Concatenate</span><span class="p">,</span> <span class="n">Embedding</span><span class="p">,</span> <span class="n">Input</span>
<span class="kn">from</span> <span class="nn">keras.models</span> <span class="kn">import</span> <span class="n">Model</span>
<span class="n">shapes</span> <span class="o">=</span> <span class="n">bg</span><span class="o">.</span><span class="n">shape</span>
<span class="n">n_classes</span> <span class="o">=</span> <span class="n">bg</span><span class="o">.</span><span class="n">n_classes</span>
<span class="n">var1_input</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="n">shapes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span>
<span class="n">var2_input</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="n">shapes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">2</span><span class="p">])</span>
<span class="n">var1_emb</span> <span class="o">=</span> <span class="n">Embedding</span><span class="p">(</span><span class="n">n_classes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="mi">10</span><span class="p">)(</span><span class="n">var1_input</span><span class="p">)</span>
<span class="n">var2_emb</span> <span class="o">=</span> <span class="n">Embedding</span><span class="p">(</span><span class="n">n_classes</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">],</span> <span class="mi">10</span><span class="p">)(</span><span class="n">var2_input</span><span class="p">)</span>
<span class="n">features</span> <span class="o">=</span> <span class="n">Concatenate</span><span class="p">()([</span><span class="n">var1_emb</span><span class="p">,</span> <span class="n">var2_emb</span><span class="p">])</span>
<span class="c1"># shapes[1] is (None, 3), so shapes[1][1] is just 3</span>
<span class="n">classes</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">shapes</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">1</span><span class="p">],</span> <span class="n">activation</span><span class="o">=</span><span class="s1">'softmax'</span><span class="p">)(</span><span class="n">features</span><span class="p">)</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="p">([</span><span class="n">var1_input</span><span class="p">,</span> <span class="n">var2_input</span><span class="p">],</span> <span class="n">classes</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s1">'adam'</span><span class="p">,</span> <span class="s1">'categorical_crossentropy'</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">fit_generator</span><span class="p">(</span><span class="n">bg</span><span class="p">)</span>
</code></pre></div>
<h1 id="installation">Installation</h1>
<p><code>shell script
pip install git+https://github.com/maxsch3/keras-batchflow.git</code></p>
</div>
</div><footer>
<div class="rst-footer-buttons" role="navigation" aria-label="Footer Navigation">
<a href="user_guide/" class="btn btn-neutral float-right" title="User guide">Next <span class="icon icon-circle-arrow-right"></span></a>
</div>
<hr/>
<div role="contentinfo">
<!-- Copyright etc -->
</div>
Built with <a href="https://www.mkdocs.org/">MkDocs</a> using a <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<div class="rst-versions" role="note" aria-label="Versions">
<span class="rst-current-version" data-toggle="rst-current-version">
<span>
<a href="https://github.com/maxsch3/keras-batchflow/" class="fa fa-github" style="color: #fcfcfc"> GitHub</a>
</span>
<span><a href="user_guide/" style="color: #fcfcfc">Next »</a></span>
</span>
</div>
<script src="js/jquery-3.6.0.min.js"></script>
<script>var base_url = ".";</script>
<script src="js/theme_extra.js"></script>
<script src="js/theme.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-MML-AM_CHTML"></script>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
<!--
MkDocs version : 1.6.1
Build Date UTC : 2024-12-01 09:46:37.978044+00:00
-->