Jekyll2022-01-15T09:03:19+00:00https://dankeder.com/feed.xmlDan KederDan Keder - freelance software engineer & system administratorDan KederDump a sample of data from PostgreSQL2016-06-05T11:00:00+00:002016-06-05T11:00:00+00:00https://dankeder.com/posts/pg-dump-sample<p>Some time ago I needed to get a small sample of data for development and testing
from a huge (production) PostgreSQL database – something like “select these 100
users and their data from other tables and dump it into a file”.</p>
<p>The standard <code class="language-plaintext highlighter-rouge">pg_dump</code> tool is not good for this. It can only dump the whole
database or just some of the tables, but not a subset of rows from particular
tables. After looking around the net for a while I didn’t find an easy way to do
it.</p>
<p>So as part of my recent efforts to learn <a href="https://golang.org/">Go</a> I created a
small tool called <a href="https://github.com/dankeder/pg_dump_sample"><code class="language-plaintext highlighter-rouge">pg_dump_sample</code></a>
that does that. You tell it which tables it should dump and how and it creates
a dump file containing the dataset. Because the manifest file which specifies
what tables should be dumped can be reused it’s a matter of running the tool
again to get a fresh dump. The resulting dump file can be loaded into
an empty database by <code class="language-plaintext highlighter-rouge">psql</code> or a similar tool. I found this approach to be a
very convenient way to keep the dev and testing environments up-to-date.</p>
<p>And the last thing - you can find <a href="https://github.com/dankeder/pg_dump_sample"><code class="language-plaintext highlighter-rouge">pg_dump_sample</code></a> on
<a href="https://github.com/dankeder/pg_dump_sample">Github</a>, along with a bit of
documentation and examples of use.</p>Dan KederSome time ago I needed to get a small sample of data for development and testing from a huge (production) PostgreSQL database – something like “select these 100 users and their data from other tables and dump it into a file”.Moved to a new domain2016-03-10T16:30:00+00:002016-03-10T16:30:00+00:00https://dankeder.com/posts/new-domain<p>I decided to move this site to a new domain
<a href="https://dankeder.com/">dankeder.com</a>. In case you have my old domain
(keder.me) in bookmarks or in the feed reader please change it to
<a href="https://dankeder.com/">dankeder.com</a>.</p>
<p>From time to time, this site also serves as a “guinea pig” for some of my
experiments. As a result, from now on it is served by
<a href="https://cloudflare.com/">CloudFlare</a>, it is HTTPS-only and even its DNS records
are secured by DNSSEC – not even some banks have this, right?</p>
<p>I also revamped the title page to be more “<a href="https://dankeder.com">professional</a>”.
Check it out if you are looking for a freelancing “DevOps engineer”. Who knows, we might
end up working together.</p>Dan KederI decided to move this site to a new domain dankeder.com. In case you have my old domain (keder.me) in bookmarks or in the feed reader please change it to dankeder.com.Recovering data with ddrescue2015-11-13T16:40:00+00:002015-11-13T16:40:00+00:00https://dankeder.com/posts/ddrescue<p>I had a hard-drive failure some time ago. The drive was in a really bad shape,
which meant that restoring the data in place is out of question.
I needed a relatively safe way to dump the disk image to another drive and
restore the filesystem there. The usual <code class="language-plaintext highlighter-rouge">dd(1)</code> did not work because of a huge
number of read errors. I found <code class="language-plaintext highlighter-rouge">ddrescue(1)</code>, which is doing the same thing
as <code class="language-plaintext highlighter-rouge">dd</code> but it’s way more tolerant to read errors.</p>
<p>After some time I came up with this combo:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue -fv -c 32 -r 1 -n /dev/sda3 /dev/sdc1 ddrescue-sda3-recovery.log
</code></pre></div></div>
<p>Let’s dissect it and see what it does:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">/dev/sda3</code> – Input device it reads data from</li>
<li><code class="language-plaintext highlighter-rouge">/dev/sdc1</code> – Output device it writes data to</li>
<li><code class="language-plaintext highlighter-rouge">ddrescue-sda3-recovery.log</code> – log file that can be used to resume an interrupted dump later</li>
<li><code class="language-plaintext highlighter-rouge">-n</code> – skip the scrape phase</li>
<li><code class="language-plaintext highlighter-rouge">-r 1</code> – how many times will the program try to read the drive if there were some read errors.</li>
<li><code class="language-plaintext highlighter-rouge">-c 32</code> – cluster size, the number of sectors to copy at a time</li>
<li><code class="language-plaintext highlighter-rouge">-f</code> – force overwrite of the output device/file</li>
<li><code class="language-plaintext highlighter-rouge">-v</code> – verbose mode</li>
</ul>
<p>There are many more options, which are documented in the manual page. There are
no examples of use though.</p>
<p>After a couple of hours or days we get the dump that we can use to restore data.
using other tools. In my case I was able to restore part of the filesystem
just with <code class="language-plaintext highlighter-rouge">fsck.ext4(8)</code>. But as usual, YMMV.</p>
<p>Beware: Don’t just blindly copy/paste the command above. Always double-check the
devices and files you read from/to so you don’t destroy your data!</p>Dan KederI had a hard-drive failure some time ago. The drive was in a really bad shape, which meant that restoring the data in place is out of question. I needed a relatively safe way to dump the disk image to another drive and restore the filesystem there. The usual dd(1) did not work because of a huge number of read errors. I found ddrescue(1), which is doing the same thing as dd but it’s way more tolerant to read errors.Build Tools, Not Solutions2015-08-21T11:48:00+00:002015-08-21T11:48:00+00:00https://dankeder.com/posts/build-tools-not-solutions<p>A couple of months ago I was on a conference in Berlin (IIRC it was <a href="http://www.erlang-factory.com/">Erlang
Factory</a>). After that much time, almost everything I
saw there has already evaporated out of my mind. Except one thing, that has been
lingering in the back of my mind since then. It’s a quote from the presentation
of Robert Virding, one of the designers of the Erlang language:</p>
<blockquote>
<p>Build tools, not solutions.</p>
</blockquote>
<!--more-->
<p>Solutions are single-purpose - they usually only solve one problem (or one instance
of the problem), and that’s it. If you want to solve something similar, you have
to get (and pay for) a new “solution”. Tools are more universal - they are
flexible enough to be used in unexpected ways, many of which were not
anticipated by the author.</p>Dan KederA couple of months ago I was on a conference in Berlin (IIRC it was Erlang Factory). After that much time, almost everything I saw there has already evaporated out of my mind. Except one thing, that has been lingering in the back of my mind since then. It’s a quote from the presentation of Robert Virding, one of the designers of the Erlang language: Build tools, not solutions.Automating DB schema updates2015-03-02T16:00:00+00:002015-03-02T16:00:00+00:00https://dankeder.com/posts/database-migrations-sqitch<p>Some time ago I’ve been searching for a tool for automating PostgreSQL database
updates. You know the drill - creating new tables, adding columns, renaming
stuff, etc. Doing these things by hand is error prone and cumbersome, and
sometimes not even possible, especially if you have more than one database or if
you need to share your changes with more developers.</p>
<!--more-->
<p>Add to that the distributed
nature of development these days and you can have a lot of fun trying to figure
out what a column is supposed to be called like after three different developers
renamed it at the same time.</p>
<p>Apparently, there are a
<a href="http://flywaydb.org/">lot</a>
<a href="https://alembic.readthedocs.org/en/latest/">of</a>
<a href="http://www.liquibase.org/">tools</a>
out there trying to solve the problem of “DB schema migration”. But as I
unfortunately found out after evaluating many of them, most of them did not suit
my needs:</p>
<ul>
<li>Ability to use simple <code class="language-plaintext highlighter-rouge">*.sql</code> files; no complicated ORM or XML bullshit please</li>
<li>Simple interface - I don’t want to type a kilometer-long command to get
something done</li>
<li>Database should “know” what version it is in, new changes should be applied
seamlessly</li>
<li>Ability to bring a pre-existing database under the change management</li>
<li>Ability to revert applied changes</li>
<li>Play nice with git, which means it should generate as few conflicts as
possible when merging branches and resolving these conflicts should be easy</li>
<li>The tool shouldn’t be bound to a particular programming language or framework</li>
<li>No Java</li>
</ul>
<p>So I kept on searching, until I stumbled upon <a href="http://sqitch.org/">Sqitch</a>. It
turned out that Sqitch covered most of the things in the list above.</p>
<p>Additionally, it supports many SQL databases (including PostgreSQL). It does not
assume almost anything about the workflow it’s used in, which makes it very easy
to integrate with deployment tools such as
<a href="http://www.ansible.com/how-ansible-works">Ansible</a>. Performing the actual
update of the database schema is as simple as running <code class="language-plaintext highlighter-rouge">sqitch --engine pg
deploy</code>. To learn how to use it I only had to read through the
<a href="https://metacpan.org/pod/distribution/App-Sqitch/lib/sqitchtutorial.pod">Sqitch tutorial</a> -
so “thumbs up”, another problem solved :-).</p>Dan KederSome time ago I’ve been searching for a tool for automating PostgreSQL database updates. You know the drill - creating new tables, adding columns, renaming stuff, etc. Doing these things by hand is error prone and cumbersome, and sometimes not even possible, especially if you have more than one database or if you need to share your changes with more developers.Moving to GitHub Pages2015-01-24T12:20:00+00:002015-01-24T12:20:00+00:00https://dankeder.com/posts/migration-to-github-pages<p>I decided to move this little blog of mine to <a href="https://help.github.com/articles/what-are-github-pages/">GitHub
Pages</a>. Using git to
manage blog posts it is a lot more appealing to me than copy-pasting stuff to
Tumblr.</p>
<p>If you use a RSS reader, please update <a href="https://dankeder.com/feed.xml">the feed URL</a>.
I would gladly set up a redirect, but GitHub Pages doesn’t support them. Sorry
for the inconvenience.</p>Dan KederI decided to move this little blog of mine to GitHub Pages. Using git to manage blog posts it is a lot more appealing to me than copy-pasting stuff to Tumblr.Playing with Ansible2014-10-28T10:02:19+00:002014-10-28T10:02:19+00:00https://dankeder.com/posts/playing-with-ansible<p>I’ve been working with <a href="http://www.ansible.com/">Ansible</a> for several weeks now.
It’s a very powerful tool for automating server deployment (like Chef or
Puppet). It’s very easy to use, the only thing that it expects from the server
is a working SSH access. Also it has a pretty good <a href="http://docs.ansible.com">documentation</a>.</p>
<p>Ansible is modular - in fact it uses modules for doing all the heavy lifting.
There are a lot of <a href="http://docs.ansible.com/modules.html">modules</a>, and you can
also write your own. All modules are idempotent, which means that they first
check what needs to be done and only do their thing if it’s really needed.
Troubleshooting is easy - the modules are simple and quite low-level
that you always know what is going. On the other hand, the lack of abstractions is a double-edged
sword: Ansible playbooks are usually bound to the specific environment they were
written for, e.g. you can’t just take any playbook written for Debian and use it
in Centos, because they use completely different packaging systems and Ansible
has specialized modules for installing packages - in this case <code class="language-plaintext highlighter-rouge">yum</code> and <code class="language-plaintext highlighter-rouge">apt</code>.</p>
<h2 id="what-where">What? Where?</h2>
<p>The thing that makes Ansible very powerful is the separation of “what” from
“where”. <a href="http://docs.ansible.com/playbooks.html">Playbooks</a>
specify what you want to do (e.g. install a package, generate a config
file from a template, start the service), but they don’t know where will
they actually run - because they can run anywhere. The information about “where” is stored in
<a href="http://docs.ansible.com/intro_inventory.html">inventory files</a>. They are simple
text files that contain hostnames or IP addresses of hosts where the playbooks
shall run. Entries in the inventory file can be grouped - for instance, you can
have a group for www servers and a group for loadbalancers.</p>
<p>You can have many inventory files for various use cases, for example an
inventory files for production servers and another one for virtual machines used
for development.</p>
<h2 id="real-example-well-almost">Real Example (well, almost)</h2>
<p>But this is just talking, so let’s try something real. Say we want to
deploy <a href="https://github.com/mitsuhiko/flask/tree/0.10.1/examples/flaskr">this simple web application</a>
to a virtual machine running CentOS 7 and put HAProxy in front of it. How do we
do that?</p>
<h3 id="create-a-virtual-machine">Create a virtual machine</h3>
<p>First, create a virtual machine running <a href="http://www.centos.org/download/">Centos
7</a> so we have something to work with.
Install VirtualBox or something similar and create yourself one. Also make sure you can
SSH to it with an SSH key.</p>
<h3 id="clone-the-flask-repo">Clone the Flask repo</h3>
<p>Then clone the official Flask repo, it contains our example web application
(checkout the last stable version):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/mitsuhiko/flask.git /path/to/flask.git
cd /path/to/flask.git && git checkout -b v0.10.1 0.10.1
</code></pre></div></div>
<h3 id="inventory-files">Inventory files</h3>
<p>Now we can create the deployment procedure. We will use the directory layout from
<a href="http://docs.ansible.com/playbooks_best_practices.html">Ansible Best Practices</a>,
because it makes things nicely organized and has a couple of convenient properties
like not having to type full file paths in playbooks.</p>
<p>First, we create the inventory file <code class="language-plaintext highlighter-rouge">inventory/hosts</code>. Put the IP address of
your virtual machine to it, my VM had <code class="language-plaintext highlighter-rouge">192.168.56.170</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[webservers]
192.168.56.170
[haproxy]
192.168.56.170
</code></pre></div></div>
<h3 id="playbook-files">Playbook files</h3>
<p>Now we can create the playbooks. Create the <code class="language-plaintext highlighter-rouge">site.yml</code> with the following contents:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
- include: webservers.yml
- include: haproxy.yml
</code></pre></div></div>
<p>This is a very simple Ansible playbook, it just includes two other playbooks for setting up the webserver
and haproxy. This is just a convenience file we will use later for deploying the whole “site”.</p>
<p>The contents of <code class="language-plaintext highlighter-rouge">webservers.yml</code> is more interesting:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
- hosts: webservers
remote_user: root
roles:
- webserver
</code></pre></div></div>
<p>Here we say that the playbook operates on a group of hosts called <code class="language-plaintext highlighter-rouge">webservers</code>
and fulfills the role of “webserver”.</p>
<h3 id="roles">Roles</h3>
<p>By convention, roles are stored in subdirectories in <code class="language-plaintext highlighter-rouge">roles/</code>. Tasks for
setting up the webserver role are stored in file <code class="language-plaintext highlighter-rouge">roles/webserver/tasks/main.yml</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
- name: create user
user: name="" shell="/sbin/nologin" state="present"
- name: install python-flask
yum: name="python-flask" state="present"
- name: install rsync
yum: name="rsync" state="present"
- name: copy flaskr
synchronize: src="/" dest="/" recursive="yes" delete="yes" rsync_opts="--exclude *.swp, --exclude *.swo" owner="no" group="no"
notify:
- restart flaskr
- name: generate flaskr.cfg
template: src="flaskr.cfg.j2" dest="/flaskr.cfg" owner="flaskr" group="flaskr" mode="644"
notify:
- restart flaskr
- name: generate flaskr.service
template: src="flaskr.service.j2" dest="/etc/systemd/system/flaskr.service" owner="root" group="root" mode="644"
notify:
- reload systemd
- name: start flaskr
service: name="flaskr" state="started"
</code></pre></div></div>
<p>It’s pretty self-explanatory, isn’t it? It will create a user that the app will run
under, install Flask, copy the app over, generate config and start the app. Easy.</p>
<p>Ansible can use <a href="http://jinja.pocoo.org/docs/dev/">Jinja2</a> templates to generate
config files. So no more <em>ad-hoc awk`ing</em> of config files and wondering why it
doesn’t work. Templates for the “webserver” role are usually stored in
<code class="language-plaintext highlighter-rouge">role/webserver/templates/</code>. For example, a systemd service file for the flaskr
<code class="language-plaintext highlighter-rouge">flaskr.service.j2</code> looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Flaskr - a minimal blog application
[Service]
Type=simple
Environment="FLASKR_SETTINGS=/flaskr.cfg"
ExecStart=/usr/bin/python /flaskr.py
User=
Group=
Restart=always
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<p>You can use any Jinja2 syntax there, including control flow statements.
Config files can have logic now!</p>
<p>The values of variables can come from several places, I recommend looking at
<a href="http://docs.ansible.com/playbooks_variables.html">Playbook Variables docs</a> to
get the idea. In this case we have a YAML file <code class="language-plaintext highlighter-rouge">group_vars/webservers</code> which
contains the variables for the hosts group <code class="language-plaintext highlighter-rouge">webservers</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>---
flaskr_user: flaskr
flaskr_group: flaskr
flaskr_home: /home/flaskr
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">haproxy</code> role is very similar, there’s no need to cut-n-paste it here. If
you are intersted you can find the complete example on
<a href="https://github.com/dankeder/ansible-example">Github</a>.</p>
<h3 id="thats-it">That’s it!</h3>
<p>After you have everything set, you can run the <code class="language-plaintext highlighter-rouge">site.yml</code> playbook:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ansible-playbook site.yml -i inventory/hosts -e "flaskr_dir=/path/to/flask.git/examples/flaskr/"
</code></pre></div></div>
<p>So to sum it up - I would say that the initial effort put into writing playbooks
pays off very quickly. Also you should store the deployment procedure in a git
repo, so you can easily keep track of all configuration changes.</p>Dan KederI’ve been working with Ansible for several weeks now. It’s a very powerful tool for automating server deployment (like Chef or Puppet). It’s very easy to use, the only thing that it expects from the server is a working SSH access. Also it has a pretty good documentation.Interesting Vim Plugins2014-04-30T14:29:38+00:002014-04-30T14:29:38+00:00https://dankeder.com/posts/vim-plugins<!--more-->
<h2 id="youcompleteme">YouCompleteMe</h2>
<p><a href="https://github.com/Valloric/YouCompleteMe">YouCompleteMe</a> is a fast auto-completion engine with support for many programming
languages (C/C++, Python, Ruby, PHP, …).</p>
<h2 id="ctrlp">CtrlP</h2>
<p><a href="https://github.com/kien/ctrlp.vim">CtrlP</a> is a fuzzy file finder that makes opening new files a breeze.</p>
<p>For those that want to open the file in a new tab instead of the current one
<a href="https://github.com/kien/ctrlp.vim/issues/160">here’s the magic that does that</a>.</p>
<h2 id="airline">Airline</h2>
<p><a href="https://github.com/bling/vim-airline">Airline</a> will prettify the vim status bar. It does similar things as the Powerline plugin but
is more lightweight.</p>
<h2 id="tagbar">Tagbar</h2>
<p><a href="https://github.com/majutsushi/tagbar">Tagbar</a> provides a sidebar for browsing tags of the source code files,
ordered by the scope.</p>
<h2 id="closetag">Closetag</h2>
<p><a href="http://www.vim.org/scripts/script.php?script_id=13">Closetag</a> is a plugin for closing HTML tags.</p>
<h2 id="hicursorwords">HiCursorWords</h2>
<p><a href="http://www.vim.org/scripts/script.php?script_id=4306">HiCursorWords</a> highlights other occurences of the word under the cursor without searching for
it. Very useful when searching for other occurences of a variable or function.</p>Dan KederAdding custom commands to setup.py2014-04-02T12:20:13+00:002014-04-02T12:20:13+00:00https://dankeder.com/posts/adding-custom-commands-to-setup-py<p>I’m often using <a href="http://pythonhosted.org/setuptools/">Setuptools</a> to package
and distribute Python modules. Recently I needeed to add a custom command
to <code class="language-plaintext highlighter-rouge">setup.py</code> so I can run it like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python setup.py mycommand --option --another-option value
</code></pre></div></div>
<p>The official documentation of Setuptools isn’t very specific on how to do it so
here is what I came up with after a little research.</p>
<!--more-->
<p>Long story short: We need to subclass the class <code class="language-plaintext highlighter-rouge">setuptools.Command</code> and
register it with Setuptools.</p>
<h2 id="create-the-command-class">Create the command class</h2>
<p>So we do not work with an overly artificial example, let’s make a command for
converting <code class="language-plaintext highlighter-rouge">*.svg</code> image files into PNGs. Let’s call it <code class="language-plaintext highlighter-rouge">gen_images</code>.</p>
<ul>
<li>
<p>Create the <code class="language-plaintext highlighter-rouge">GenImagesCommand</code> class by subclassing <code class="language-plaintext highlighter-rouge">setuptools.Command</code>. Put it
directly into <code class="language-plaintext highlighter-rouge">setup.py</code>.</p>
</li>
<li>
<p>Initialize the class attribute <code class="language-plaintext highlighter-rouge">user_options</code>. It is a list of tuples where
each tuple corresponds to a single command-line option. The tuple consists of
the option long name, short name (without the leading ‘–’ and ‘-‘) and
description. Options are parsed using the <code class="language-plaintext highlighter-rouge">distutils.fancy_getopt</code> module.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> user_options = [
('input-dir=', 'i', 'input directory'),
('output-dir=', 'o', 'output directory'),
]
</code></pre></div> </div>
</li>
<li>
<p>Implement the method <code class="language-plaintext highlighter-rouge">initialize_options()</code>. It is used to initialize the options to
default values.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def initialize_options(self):
self.input_dir = None
self.output_dir = None
</code></pre></div> </div>
</li>
<li>
<p>Implement the method <code class="language-plaintext highlighter-rouge">finalize_options()</code>. It is used to check final option
values. For example, you may want to check if a pathname exists, compute
missing values or process dependencies between options.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def finalize_options(self):
if self.input_dir is None:
raise Exception("Parameter --input-dir is missing")
if self.output_dir is None:
raise Exception("Parameter --output-dir is missing")
if not os.path.isdir(self.input_dir):
raise Exception("Input directory does not exist: {0}".format(self.input_dir))
if not os.path.isdir(self.output_dir):
raise Exception("Output directory does not exist: {0}".format(self.output_dir))
</code></pre></div> </div>
</li>
<li>
<p>Implement the method <code class="language-plaintext highlighter-rouge">run()</code>. The <code class="language-plaintext highlighter-rouge">run()</code> method does the “hard work” of the command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> def run(self):
def _gen_images(arg, dirname, fnames):
for fname in fnames:
if self.verbose:
print 'processing "{0}"'.format(os.path.join(dirname, fname))
# FIXME: Really process the files here
os.path.walk(self.input_dir, _gen_images, None)
</code></pre></div> </div>
</li>
<li>
<p>Optionally you can set the value of <code class="language-plaintext highlighter-rouge">description</code> class attribute. It is used to
describe what the command does when you run <code class="language-plaintext highlighter-rouge">python setup.py --help-commands</code>.</p>
</li>
</ul>
<p>Here is the complete <code class="language-plaintext highlighter-rouge">GenImagesCommand</code> class:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import os
from setuptools import Command
class MyCommand(Command):
""" Run my command.
"""
description = 'generate images'
user_options = [
('input-dir=', 'i', 'input directory'),
('output-dir=', 'o', 'output directory'),
]
def initialize_options(self):
self.input_dir = None
self.output_dir = None
def finalize_options(self):
if self.input_dir is None:
raise Exception("Parameter --input-dir is missing")
if self.output_dir is None:
raise Exception("Parameter --output-dir is missing")
if not os.path.isdir(self.input_dir):
raise Exception("Input directory does not exist: {0}".format(self.input_dir))
if not os.path.isdir(self.output_dir):
raise Exception("Output directory does not exist: {0}".format(self.output_dir))
def run(self):
def _gen_images(arg, dirname, fnames):
for fname in fnames:
if self.verbose: # verbose is provided "automagically"
print 'processing "{0}"'.format(os.path.join(dirname, fname))
# FIXME: Really process the file here
os.path.walk(self.input_dir, _gen_images, None)
</code></pre></div></div>
<h2 id="register-the-command-in-setuptools">Register the command in Setuptools</h2>
<p>Currently there are two ways to register the command class in the Setuptools
framework.</p>
<p>The first one is to add a <code class="language-plaintext highlighter-rouge">cmdclass</code> key to the <code class="language-plaintext highlighter-rouge">setup()</code> call in <code class="language-plaintext highlighter-rouge">setup.py</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> setup(
name="my-project",
cmdclass={
'gen_images': GenImagesCommand,
},
# other stuff ...
)
</code></pre></div></div>
<p>The key is the command name, the value is the command class.</p>
<p>Another way to register the command is to use the <code class="language-plaintext highlighter-rouge">entry_points</code> key.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> setup(
name="my-project",
entry_points={
'distutils.commands': [
'gen_images = package.module:GenImagesCommand',
],
},
# other stuff ...
)
</code></pre></div></div>
<p>This states that the command <code class="language-plaintext highlighter-rouge">gen_images</code> is implemented by the class
<code class="language-plaintext highlighter-rouge">GenImagesCommand</code> from the module <code class="language-plaintext highlighter-rouge">package.module</code>. The command class will be
automatically imported by Setuptools when it’s needed. However, in this case the
command class must be stored in a separate module, not directly in <code class="language-plaintext highlighter-rouge">setup.py</code>.</p>
<h2 id="running-the-command">Running the command</h2>
<p>Now is possible to run the command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python setup.py gen_images --input-dir assets/ --output-dir images/
running gen_images
processing "assets/a.svg"
processing "assets/c.svg"
processing "assets/b.svg"
</code></pre></div></div>Dan KederI’m often using Setuptools to package and distribute Python modules. Recently I needeed to add a custom command to setup.py so I can run it like this: $ python setup.py mycommand --option --another-option value The official documentation of Setuptools isn’t very specific on how to do it so here is what I came up with after a little research.How to remove ads in Android2014-03-11T06:32:37+00:002014-03-11T06:32:37+00:00https://dankeder.com/posts/get-rid-of-ads<p>This is a very simple yet effective way to get rid of advertisements
in Android apps – without rooting the device. In principle it is the
same method that the Android AdBlock app uses: resolve all DNS queries for
various ad servers to <code class="language-plaintext highlighter-rouge">127.0.0.1</code>. But AdBlock app requires a rooted device, which
not everyone is able or wants to do.</p>
<p>So instead of mocking DNS queries on the phone or tablet we do it on the “next
hop” device - i.e. home or company router. That will affect <em>all</em> devices
connected to that network - all phones, tablets and computers. But this method
also has a drawback - you need a capable router, ideally with root SSH access to
it. What I use (and recommend) is a Raspberry PI with Raspbian.</p>
<p>Just put the following simple script into <code class="language-plaintext highlighter-rouge">/etc/hosts.d/update.sh</code>, make it executable and run it. You
may have to create the directory <code class="language-plaintext highlighter-rouge">/etc/hosts.d</code> first. You can also run the script in cron so everything updates regularly.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/bin/bash
curl 'http://adaway.sufficientlysecure.org/hosts.txt' > /etc/hosts.d/01hosts
curl 'http://winhelp2002.mvps.org/hosts.txt'> /etc/hosts.d/02hosts
curl 'http://hosts-file.net/.\ad_servers.txt'> /etc/hosts.d/03hosts
curl 'http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext' > /etc/hosts.d/04hosts
</code></pre></div></div>
<p>Raspbian uses <code class="language-plaintext highlighter-rouge">dnsmasq</code> to handle DNS queries so we need to tell it to consult
the files in <code class="language-plaintext highlighter-rouge">/etc/hosts.d</code> before contacting the real DNS servers. To do it add
the following lines into <code class="language-plaintext highlighter-rouge">/etc/dnsmasq.conf</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Disable ads in android apps
addn-hosts=/etc/hosts.d/01hosts
addn-hosts=/etc/hosts.d/02hosts
addn-hosts=/etc/hosts.d/03hosts
addn-hosts=/etc/hosts.d/04hosts
</code></pre></div></div>
<p>And restart the dnsmasq service:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/init.d/dnsmasq restart
</code></pre></div></div>
<p>And that’s it, now you shouldn’t see any of the annoying ads anymore.</p>Dan KederThis is a very simple yet effective way to get rid of advertisements in Android apps – without rooting the device. In principle it is the same method that the Android AdBlock app uses: resolve all DNS queries for various ad servers to 127.0.0.1. But AdBlock app requires a rooted device, which not everyone is able or wants to do.