Alteroot:~#

New blog engine!

Fri, 12 Sep 2025 00:00:00 +0000

Ok, the last article was 5 years ago, setup was 12 years old, time for a revamp!

I switched from jekyllrb to gohugo, with a new “modern” theme, and imported all articles. URLs should be stay same.

I hope LLM scrapper will love it ;-)

Maybe I’ll write some new posts, I’m still kinda involved in Xfce (on the infra side), and I work again with Apache Cassandra at $job, so… :)

Xfce infrastructure in 2020

Sat, 18 Jul 2020 10:21:00 +0200

Oh boy, 3 years without any blogpost ! So much things happened in the Xfce world!
First, we finally released Xfce 4.14 in August 2019, more than 4 years of effort, a migration to gtk3, and so much new things, but you already know that, and you surely already run it. We are now working on Xfce 4.16 , with the hope to release it this year (yes, in 2020) !
You can read more development news on our blog.
Some buzzwords: new icons, CSD, drop of gtk2 support, UI revamp, and tons of other stuff.

I want to talk a bit about the infrastructure.
Last year, we had a Gandi sponsorship for our servers. I started an Ansible repository with all our services (no more -undocumented- local modifications!), and we migrated a big part of our infra to the new servers (docs, www, wiki etc).

Then came the time to think about our development environment. We used the cgit/gitolite/bugzilla triptych, which worked fine for us for more than a decade, but make difficulty to have new contributor (our workflow was…hm, suboptimal).
We tested differents solutions (mainly Gitea and GitLab), and we opted for the famous Gitlab, already used by multiples open source projects. In the core team we were several to use it at work, so we were in a familiar environment.

For months, a gitlab was running on one of our Gandi instance, to do tests and prepare the transition.

But it was so much work that we never started the real migration. Then came the Covid-19 lockdown. No friends, no bars, lot of free time, so why not spent it on Xfce ?

With Simon (ochosi) and Andre (andreldm), we started to use our own gitlab by opening issues, creating todo, doing merge requests etc related to the migration.
The 1st of May, we finally did the migration to GitLab ! Woot !

Jason (j4yav), working for Gitlab inc, joined us and proposed to help us for the Gitlab CI/CD setup. At the same time, we got a new (sponsored !) VM from FossHost ! Perfect timing, we can now run a dedicated gitlab-runner instance, and we started to use gitlab-ci !

As of July 2020, almost all Xfce components (core, apps, panel plugins) run make distcheck on every merge requests and push to master, thanks to our xfce-build docker container.

Now that we have a beautiful GitLab, merge requests and CI, the only missing part was… the migration from Bugzilla !
I forked and modified bztogl, a tool developped and used by Gnome and Freedesktop folks when they migrated away from bugzilla to gitlab in 2018/2019.
Initially, we wanted to migrate all bugzilla issues to gitlab (opened and closed ones). But we had more than 15000 issues ! We decided to only migrate opened issues. We took the opportunity to do a biiig cleanup of old bugs by closing more than 500 of them.

A important condition of the bugzilla migration was to not lose our 15 years of history.
AndreLDM did a huge work by creating bzstatic, a tool to snapshot a bugzilla instance and make it a simple static html version : https://bugzilla.xfce.org.
He open sourced it at https://gitlab.xfce.org/andreldm/bzstatic !

As a result, we now have a modern developments workflow, fully integrated with gitlab. We enabled github and gitlab.com auth on our instance (some people requested it), and since 3 months, we’ve seen a surge of new contributors, issues and merge requests ! Really a good news for the Xfce project ! We also had a sponsorship from Scaleway lately, that we need to set up !

So much things left to do, like update all our documentations for the new workflow, work on Xfce 4.16, get rid of our old release manager app and use gitlab-ci to generate artifacts, simplify our translation workflow and tons of other things !

Enjoy !

Road to Xfce 4.14, part 2

Tue, 30 May 2017 18:00:00 +0000

More than 1 year since my latest blogpost… wow !
Lots of activities these last months, on code and infra side :)
But the good news is that we had done a lots of background tasks (see the Infrastructure section) !

As you might know, Xfce 4.14 is a transitionnal release, where the main goal is to port all components to GTK+3.

GTK+3 minimum version has been bumped to version 3.20 (and maybe 3.22), because of so much changes since gtk 3.14 (broken themes, css), this will remove a lot of ifdef maze and code complexity to support several gtk versions. By the time of 4.14 release, all major distributions will have gtk>3.20.

You can check the status of all currents efforts on the wiki. Please check this page before starting a GTK+3 port, maybe it’s already done but not yet on master !

Core

Almost all core components have now a gtk+3 version (yai !), with developement releases lately ! Here a small summary :

In the help wanted section, AndreLDM is currently working on Thunar gtk+3 port ! If you have gtk/gdk/cairo knowledges, any contributor is welcome ! Contact us on #xfce-dev irc channel (freenode) or xfce4-dev mailist.

Ochosi made some great blog posts on all activities theses last months, (xfce4-notifyd, panel, taskmanager, clipman), stay tuned on https://blog.xfce.org/ !

Xfce Apps

In the xfce apps world, please notice the work on xfce4-terminal , with a complete port to gtk+3 in 0.8 (+ get rid of the old and insecure vte 0.28), new features and more : https://git.xfce.org/apps/xfce4-terminal/tree/NEWS .

This work will be available in the future Debian Stretch! Thanks to Igor for his hard work !

You can also notice the GTK+3 port of xfce4-screenshooter by AndreLDM.

Xfce Panel plugins

On the panel plugins side, tons of new releases, I can’t list all of them - netload, clipman, datetime, cpufreq etc - with of course a bunch of gtk3 ports, thanks to all of our contributors.

Look at the https://mail.xfce.org/pipermail/xfce-announce/ mailist to see the work done.

I also updated the xfce4-sample-plugin which can be used as a base if you want to start a new panel plugin !

Tests

If you want to test the current state of Xfce 4.14, you can look at xfce-test, a docker image by Florian which allow you to test git master and to run behave (behaviour-driven development) ! -okay sometimes it’s b0rken-

Ochosi wrote a small blog post about this docker image last month.

Here a screenshot of a git master build (a.k.a : xfce 4.14) :

Infrastructure

Almost all the infrastructure problems we had by the time of my last blog post have been fixed.

Thanks to gaston, user repositories are now working, and all services are now HTTPS enabled (with let’s encrypt) ! Youhou !

Moreover, since a few month, I try to take care of the Xfce servers infrastructure. Here a small summary of latest actions.

Git / Bugzilla

Enabling syntax highlighting on cgit web interface ! Based on python fragment, it detect c/h files, see https://git.xfce.org/xfce/xfce4-power-manager/tree/panel-plugins/power-manager-plugin/power-manager-button.c as an example.
Bug number catch in commit message to highlight them in cgit and link to the related bugzilla issue ! It catch several patterns like Bug #xxxx or bug xxxx. Developers, please use it ! :)
Speaking of bug number, a new git hook is in place, it parse the git commit message, detect if there is a bug number referenced, and automatically posts commits references to bugzilla ! An example: https://bugzilla.xfce.org/show_bug.cgi?id=12159#c4 ! This is a really cool new feature (thanks ochosi for the idea). It’s not perfect and it need more works, but it is now available on all repositories !
The famous “Bug 12117 - The default desktop startup screen causes damage to monitor!” got a lot of attention (x10 in traffic!) last March on reddit, hackernews, fefes.de and killed our poor bugzilla instance ! (thanks mod_perl :] ). Here are some visitor stats, see the difference…
A new bugzilla template/theme by knome is available, with a lot of improvements ! You can enable it in your account preferences. It’s still a work in progress but should be released as the default theme soon
Upgrade to latest cgit version (1.1)

Website

Maybe you already seen it, but the https://xfce.org website got a nice revamp with a more modern code/css, and is now responsive. Many thanks to knome, you can read his blog post which contain more informations :)

Misc

http://geoip.xfce.org (used by the weather plugin to auto-detect the user localization) has been updated and should be more accurate. It is also available in https.
The wiki engine (dokuwiki) and plugins used on wiki.x.o, docs.x.o and goodies.x.o, has been updated to latest release.
The forum engine has been updated.
Some obsoletes/outdated subdomains now redirect to the main xfce.org website (xfc.x.o, squeeze.x.o, thunar.x.o, foundation.x.o etc)
The wiki now use an invisible captcha and not an ugly one like previously.
Tons of Infrastructure bugs opened on our bugzilla has been fixed.

Enjoy,

Road to Xfce 4.14

Tue, 12 Apr 2016 13:00:00 +0000

The looong road to xfce 4.14. I hope you still want some news about Xfce !
As usual, the xfce development is slow, has some organisation problems (only few members have access to servers, and are often not present), and lack “core” maintainers/mentors. But hey, we do our best :).

The bigger change for 4.14 is the port / upgrade to GTK+ 3.x.
It will be a like-for-like port. New and “big” features are postponed to 4.16.

You can read the thread on the ML https://mail.xfce.org/pipermail/xfce4-dev/2015-February/031146.html about the decision of only port to gtk3 and not introduce any new features for now.

The roadmap is available on the Wiki : http://wiki.xfce.org/releng/4.14/roadmap. The minimum requiered version has been fixed to be GTK+ 3.14.

Roadmap / Planned Features :

All components of -core will be ported to Gtk+ 3.
Replace dbus-glib with GDbus.
Review icon-names in all components and use consistent naming, following the fd.o spec where possible
Use symbolic icons for panel plugins and in apps where suitable
Replace deprecated widgets.

Here is a quick (and somewhat incomplete) overview of what is already done !

Core components

Work is on progress for core components, with “users” branches availables on http://git.xfce.org/ (xfce4-panel, libxfce4ui, xfce4-session, panel etc). Nothing really available for daily use though.

Lots of works on xfwm4’ compositor (in master), with implementation of vsync using OpenGL, libepoxy support, adding of DRI3/Present to the compositor, use of GLX for compositing instead of Xrender…

Libxfce4ui 4.12 already compile with gtk2 and gtk3 support, exo git master have a gtk3 preliminary support, garcon is already ported. xfdesktop have a gtk3 branch

The panel support GTK-3 plugins since 4.12, but it core need to be upgraded to use gtk3.

Xfce4-power-manager

One of the most active project. Version 1.6 (released last month) is a complete port to gtk3 and gdbus, see http://git.xfce.org/xfce/xfce4-power-manager/tree/NEWS

Xfce4-notifyd

A GTK3 branch exist, you can read the thread on the ML : https://mail.xfce.org/pipermail/xfce4-dev/2015-October/031504.html. I use it since few months, and it works pretty well :)

Xfce4-Appfinder

xfce4-appfinder progress to gtk3 : https://mail.xfce.org/pipermail/xfce4-dev/2015-August/031438.html
Can you guess which one use GTK3 ? (hint: the left one ;))

Plugins

Some plugins are still active, and are already ported to GTK-3 (xfce4-indicator-plugin, xfce4-pulseaudio-plugin), as xfce4-panel support both gtk2 and gtk3 plugins.

Last months have seen new versions of xfce4-equake-plugin, xfce4-embed-plugin, xfce4-verve-plugin, xfce4-notes-plugin, xfce4-cpufreq-plugin.

One of the more usefull new plugin is the pulseaudio-plugin, a replacement for xfce4-mixer : http://git.xfce.org/panel-plugins/xfce4-pulseaudio-plugin/, I use it and it works well.

Stay tuned !

Compte rendu du Meetup ArchLinux France du 10 Nov. 2015

Wed, 11 Nov 2015 21:30:00 +0000

Ce 10 Novembre 2015 a eu lieu le second meetup ArchLinux France, chez BlaBlaCar à Paris ! Pour rappel le premier Meetup avait eu lieu en Octobre 2014. Pour cette nouvelle édition, il y avait 5 présentations, et près de 70 personnes ! (vous pouvez retrouver le programme sur meetup).

Voici la liste des présentations et leurs slides (update: elles sont maintenant toutes disponibles !) :

Arch Linux Security Team Genesis (RbN) / Slides
Arch Linux Security Projects (Remi Gacogne) / Slides
Vous reprendrez bien un peu de yaourt? (Skunnyk) / Slides
Ansible 101 (Julien Girardin) / Slides
Arch Linux Archive / agetpkg (Sebastien Luttringer) / Slides

J’ai pour ma part parlé de Yaourt, son histoire, son développement, ses features. Si vous souhaitez participer, n’hésitez pas à vous rendre sur le github du projet :)

Mes slides ci dessous :

Et quelques photos, tirées du ‘Live Tweet’ de @archlinuxfr :

Le grand jour ! #archlinux #meetup @BlaBlaCar_FR pic.twitter.com/edg5DaM2ye
— Skunnyk (@_Skunnyk_) November 10, 2015

Et maintenant le Hardening de #archlinux #meetup https://t.co/PMO0U3DsTB pic.twitter.com/AeXvYkiMlK
— archlinuxfr (@archlinuxfr) November 10, 2015

Et pour finir, @seblu42 nous parle de 'Arch Linux Archive' ! https://t.co/IcVOm30AAB #archlinux #meetup pic.twitter.com/gNxYsUU3IQ
— archlinuxfr (@archlinuxfr) November 10, 2015

Encore merci aux talkers, à tous ceux présents, et à BlaBlaCar (et Puckel_) pour le sponsoring de la salle et la collation !

Enhance Apache Cassandra Logging

Tue, 22 Sep 2015 13:00:00 +0000

Cassandra usually output all its logs in a system.log file. It uses log4j old 1.2 version for cassandra 2.0, and since 2.1, logback, which of course use different syntax :)
Logs can be enhanced with some configuration. These explanations works with Cassandra 2.0.x and Cassandra 2.1.x, I haven’t tested others versions yet.

I wanted to split logs in different files, depending on their “sources” (repair, compaction, tombstones etc), to ease debugging, while keeping the system.log as usual.

For example, to declare 2 new files to handle, say Repair and Tombstones logs :

Cassandra 2.0 :

You need to declare each new log files in log4j-server.properties file.

[...]
## Repair
log4j.appender.Repair=org.apache.log4j.RollingFileAppender
log4j.appender.Repair.maxFileSize=20MB
log4j.appender.Repair.maxBackupIndex=50
log4j.appender.Repair.layout=org.apache.log4j.PatternLayout
log4j.appender.Repair.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n
## Edit the next line to point to your logs directory
log4j.appender.Repair.File=/var/log/cassandra/repair.log

## Tombstones
log4j.appender.Tombstones=org.apache.log4j.RollingFileAppender
log4j.appender.Tombstones.maxFileSize=20MB
log4j.appender.Tombstones.maxBackupIndex=50
log4j.appender.Tombstones.layout=org.apache.log4j.PatternLayout
log4j.appender.Tombstones.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n
### Edit the next line to point to your logs directory
log4j.appender.Tombstones.File=/home/log/cassandra/tombstones.log

Cassandra 2.1 :

It is in the logback.xml file.


  <appender name="Repair" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/repair.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>20</maxIndex>
    </rollingPolicy>

    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>20MB</maxFileSize>
    </triggeringPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
      <!-- old-style log format
      <pattern>%5level [%thread] %date{ISO8601} %F (line %L) %msg%n</pattern>
      -->
    </encoder>
  </appender>

  <appender name="Tombstones" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${cassandra.logdir}/tombstones.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${cassandra.logdir}/tombstones.log.%i.zip</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>20</maxIndex>
    </rollingPolicy>

    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>20MB</maxFileSize>
    </triggeringPolicy>
    <encoder>
      <pattern>%-5level [%thread] %date{ISO8601} %F:%L - %msg%n</pattern>
      <!-- old-style log format
      <pattern>%5level [%thread] %date{ISO8601} %F (line %L) %msg%n</pattern>
      -->
    </encoder>
  </appender>

Now that theses new files are declared, we need to fill them with logs. To do that, simply redirect some Java class to the good file. To redirect the class org.apache.cassandra.db.filter.SliceQueryFilter, loglevel WARN to the Tombstone file, simply add :

Cassandra 2.0 :

log4j.logger.org.apache.cassandra.db.filter.SliceQueryFilter=WARN,Tombstones

Cassandra 2.1 :

<logger name="org.apache.cassandra.db.filter.SliceQueryFilter" level="WARN">
    <appender-ref ref="Tombstones"/>
</logger>

It’s a on-the-fly configuration, so no need to restart Cassandra !
Now you will have dedicated files for each kind of logs.

A list of interesting Cassandra classes :

org.apache.cassandra.service.StorageService, WARN : Repair
org.apache.cassandra.net.OutboundTcpConnection, DEBUG : Repair (haha, theses fucking stuck repair)
org.apache.cassandra.repair, INFO : Repair
org.apache.cassandra.db.HintedHandOffManager, DEBUG : Repair
org.apache.cassandra.streaming.StreamResultFuture, DEBUG : Repair 
org.apache.cassandra.cql3.statements.BatchStatement, WARN : Statements
org.apache.cassandra.db.filter.SliceQueryFilter, WARN : Tombstones

You can find from which java class a log message come from by adding “%c” in log4j/logback “ConversionPattern” :

org.apache.cassandra.db.ColumnFamilyStore INFO  [BatchlogTasks:1] 2015-09-18 16:43:48,261 ColumnFamilyStore.java:939 - Enqueuing flush of batchlog: 226172 (0%) on-heap, 0 (0%) off-heap
org.apache.cassandra.db.Memtable INFO  [MemtableFlushWriter:4213] 2015-09-18 16:43:48,262 Memtable.java:347 - Writing Memtable-batchlog@1145616338(195.566KiB serialized bytes, 205 ops, 0%/0% of on/off-heap limit)
org.apache.cassandra.db.Memtable INFO  [MemtableFlushWriter:4213] 2015-09-18 16:43:48,264 Memtable.java:393 - Completed flushing /home/cassandra/data/system/batchlog/system-batchlog-tmp-ka-4267-Data.db; nothing needed to be retained.  Commitlog position was ReplayPosition(segmentId=1442331704273, position=17281204)

You can disable “additivity” (i.e avoid adding messages in system.log for example) in log4j for a specific class by adding :

log4j.additivity.org.apache.cassandra.db.filter.SliceQueryFilter=false

For logback, you can add additivity=“false” to <logger .../> elements.

To migrate from log4j logs to logback.xml, you can look at http://logback.qos.ch/translator/

Sources :

Note: you can add http://blog.alteroot.org/feed.cassandra.xml to your rss aggregator to follow all my Cassandra posts :)

Working OCSP Stapling setup with free Wosign Certificate on Nginx

Fri, 17 Jul 2015 15:00:00 +0000

Chinese CA WoSign issues free multi-domain SSL certificates valid for 3 years. You can read some informations on how to get theses free certificates on https://www.ohling.org/blog/2015/02/wosign-free-2y-ssl-certificate.html.
FYI, I received my certificate around 1hour after submission :)

Now, the nginx ssl configuration ! I won’t cover the pure SSL configuration, you can look at mozilla wiki for a “perfect setup”.
Standard procedure, you extract the provided zip, grab the 1_mydomain_bundle.crt and add it to nginx ssl configuration.

I wanted to enable OCSP stapling, for checking the revocation status of X.509 digital certificates on server side.
Basic OCSP checking potentially impairs users’ privacy and slows down browsing, since it requires the client to contact a third party (the CA) to confirm the validity of each certificate that it encounters. With OCSP Stapling, the certificate holder (your servers) queries the OCSP server themselves at regular intervals, obtaining a signed time-stamped OCSP response.
When the site’s visitors attempt to connect to the site, this response is included with the TLS Handshake.

In nginx, the stapling configuration is simple :

ssl_stapling on;
ssl_stapling_verify on;

Then you need to reload nginx. You can now test OCSP response with openssl.

openssl s_client -connect blog.alteroot.org:443 -tls1_2 -tlsextdebug -status

Note: the FIRST client request always has a empty OSCP response, because nginx query the remote OCSP server at the same time. This applies to ALL nginx workers, Nginx uses a per-worker cache of OCSP responses.

Unfortunately, it does not works, and I can see in nginx logs this error :

2015/07/17 11:31:21 [error] 19557#0: OCSP responder sent invalid "Content-Type" header: "text/html" while requesting certificate status, responder: ocsp6.wosign.com

By checking http://ocsp6.wosign.com/ (504 bad gateway), I thought that their OCSP Server was down.
After some investigations, I found this nginx ticket : You need at least Nginx 1.7.4 which fix a bug with uppercase on ngx_escape_uri() !

As I just set up a new server with debian Jessie, I used the default Nginx (1.6.2). So I need to upgrade to 1.8.0 with Dotdeb packages (thanks @gui ;)).

After upgrade to 1.8.0, in Nginx logs :

2015/07/17 11:47:30 [error] 21390#21390: OCSP_basic_verify() failed (SSL: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:Verify error:unable to get local issuer certificate) while requesting certificate status, responder: ocsp6.wosign.com

Yes ! At least nginx can talk to the OCSP server :)

Now, I need to add “trusted” certificate to Nginx.
You need to “cat” StartCom Certification Authority certificate then Wosign one.
Why Startcom ? Because the WoSign root CA is cross-signed by the StartCom CA so it is trusted by every browsers.

The StartCom Certification Authority certificate (Serial 1, PEM format) : https://ssl-tools.net/certificates/1cf30e1-startcom-certification-authority
And WoSign CA certificates (includes in the 1_mydomain_bundle.crt)

The final ssl_trusted_certificate file :

Now the OCSP check works !

$ openssl s_client -connect blog.alteroot.org:443 -tls1_2 -tlsextdebug -status
[...]
OCSP response: 
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = CN, O = WoSign CA Limited, CN = WoSign Free SSL OCSP Responder(G2)
    Produced At: Jul 17 11:12:45 2015 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: A06661F16CBCC23E98BC71914830B85AAA8D0A6B
      Issuer Key Hash: D2A716207CAFD9959EEB430A19F2E0B9740EA8C7
      Serial Number: 5FA7F414BCB7C7374B525EEB4D894072
    Cert Status: good
    This Update: Jul 17 11:12:45 2015 GMT
    Next Update: Jul 19 11:12:45 2015 GMT
[...]

Enjoy :)

Sources:

Awesome Weekly Newsletters !

Thu, 21 May 2015 15:00:00 +0000

Like the well known awesome-awesomeness repository on Github, below, you’ll find a list of Weekly (some are not so weekly ^^) newsletters.

If you want to add an unlisted newsletter, feel free to submit a PR.

Development

Python Weekly

http://www.pythonweekly.com/
Past issues: Archive

http://www.pycoders.com/
Past issues: Archive

Ruby Weekly

http://rubyweekly.com/
Past issues: Archive

http://golangweekly.com/
Past issues: Archive

DBs

DB Weekly

http://dbweekly.com/
Past issues: Archive

http://mysqlnewsletter.com/
Past issues: Archive

http://postgresweekly.com/
Past issues: Archive

NoSQL Weekly

http://www.nosqlweekly.com/

(big)Data Science

http://www.bigdataweekly.com/
Past issues: Archive

http://www.datascienceweekly.org/
Past issues: Archive

Web Development

HTML5 Weekly

http://html5weekly.com/
Past issues: Archive

JavaScript Weekly

http://javascriptweekly.com/
Past issues: Archive

Mobile Web Weekly

http://mobilewebweekly.co/
Past issues: Archive

Node Weekly

http://nodeweekly.com/
Past issues: Archive

Others

RasPi Weekly

http://www.raspiweekly.com/
Past issues: Archive

Bitcoin Weekly

http://bitcoinweekly.com/
Past issues: Archive

Enjoy !

How to change Cassandra compaction strategy on a production cluster

Mon, 20 Apr 2015 12:00:00 +0000

I’ll talk about changing Cassandra CompactionStrategy on a live production Cluster.
First of all, an extract of the Cassandra documentation :

Periodic compaction is essential to a healthy Cassandra database because Cassandra does not insert/update in place. As inserts/updates occur, instead of overwriting the rows, Cassandra writes a new timestamped version of the inserted or updated data in another SSTable. Cassandra manages the accumulation of SSTables on disk using compaction. Cassandra also does not delete in place because the SSTable is immutable. Instead, Cassandra marks data to be deleted using a tombstone.

By default, Cassandra use SizeTieredCompactionStrategyi (STC). This strategy triggers a minor compaction when there are a number of similar sized SSTables on disk as configured by the table subproperty, 4 by default.

Another compaction strategy available since Cassandra 1.0 is LeveledCompactionStrategy (LCS) based on LevelDB.
Since 2.0.11, DateTieredCompactionStrategy is also available.

Depending on your needs, you may need to change the compaction strategy on a running cluster. Change this setting involves rewrite ALL sstables to the new strategy, which may take long time and can be cpu / i/o intensive.

I needed to change the compaction strategy on my production cluster to LeveledCompactionStrategy because of our workflow : lot of updates and deletes, wide rows etc.
Moreover, with the default STC, progressively the largest SSTable that is created will not be compacted until the amount of actual data increases four-fold. So it can take long time before old data are really deleted !

Note: You can test a new compactionStrategy on one new node with the write_survey bootstrap option. See the datastax blogpost about it.

The basic procedure to change the CompactionStrategy is to alter the table via cql :

cqlsh> ALTER TABLE mykeyspace.mytable  WITH compaction = { 'class' :  'LeveledCompactionStrategy'  };

If you run alter table to change to LCS like that, all nodes will recompact data at the same time, so performances problems can occurs for hours/days…

A better solution is to migrate nodes by nodes !

You need to change the compaction locally on-the-fly, via the JMX, like in write_survey mode.
I use jmxterm for that. I think I’ll write articles about all theses jmx things :)
For example, to change to LCS on mytable table with jmxterm :

~ java -jar jmxterm-1.0-alpha-4-uber.jar --url instance1:7199                                                      
Welcome to JMX terminal. Type "help" for available commands.
$>domain org.apache.cassandra.db
#domain is set to org.apache.cassandra.db
$>bean org.apache.cassandra.db:columnfamily=mytable,keyspace=mykeyspace,type=ColumnFamilies
#bean is set to org.apache.cassandra.db:columnfamily=mytable,keyspace=mykeyspace,type=ColumnFamilies
$>get CompactionStrategyClass
#mbean = org.apache.cassandra.db:columnfamily=mytable,keyspace=mykeyspace,type=ColumnFamilies:
CompactionStrategyClass = org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy;
$>set CompactionStrategyClass "org.apache.cassandra.db.compaction.LeveledCompactionStrategy" 
#Value of attribute CompactionStrategyClass is set to "org.apache.cassandra.db.compaction.LeveledCompactionStrategy"

A nice one-liner :

~ echo "set -b org.apache.cassandra.db:columnfamily=mytable,keyspace=mykeyspace,type=ColumnFamilies CompactionStrategyClass org.apache.cassandra.db.compaction.LeveledCompactionStrategy" | java -jar jmxterm-1.0-alpha-4-uber.jar --url instance1:7199

On next commitlog flush, the node will start it compaction to rewrite all it mytable sstables to the new strategy.

You can see the progression with nodetool :

~ nodetool compactionstats
pending tasks: 48
compaction type        keyspace           table       completed           total      unit  progress
Compaction        mykeyspace       mytable      4204151584     25676012644     bytes    16.37%
Active compaction remaining time :   0h23m30s

You need to wait for the node to recompact all it sstables, then change the strategy to instance2, etc.
The transition will be done in multiple compactions if you have lots of data. By default new sstables will be 160MB large.

you can monitor you table with nodetool cfstats too :

~ nodetool cfstats mykeyspace.mytable
[...]
Pending Tasks: 0
	Table: sort
	SSTable count: 31
	SSTables in each level: [31/4, 0, 0, 0, 0, 0, 0, 0, 0]
[...]

You can see the 31/4 : it means that there is 31 sstables in L0, whereas cassandra try to have only 4 in L0.

Taken from the code ( src/java/org/apache/cassandra/db/compaction/LeveledManifest.java )

[...]
// L0: 988 [ideal: 4]
// L1: 117 [ideal: 10]
// L2: 12  [ideal: 100]
[...]

When all nodes have the new strategy, let’s go for the global alter table. /!\ If a node restart before the final alter table, it will recompact to default strategy (SizeTiered)!

~ cqlsh 
cqlsh> ALTER TABLE mykeyspace.mytable  WITH compaction = { 'class' :  'LeveledCompactionStrategy'  };

Et voilà, I hope this article will help you :)

My latest Cassandra blogpost was one year ago… I have several in mind (jmx things !) so stay tuned !

HTTP alternative services and opportunistic encryption

Sat, 28 Mar 2015 17:30:00 +0000

HTTP Alternative services allow an origin’s resources to be authoritatively available at a separate network location, possibly accessed with a different protocol configuration.
Opportunistic Encryption (OE) provides unauthenticated encryption over TLS for data that would otherwise be carried via clear text. It allows a http:// connection to use secure resources.

Update : Mozilla has disabled “opportunistic encryption” to fix HTTPS-crippling bug in Firefox 37.0.1 :(

You can read a good introduction on Mnot blog. I think this quote is a good summary :

It doesn't help at all with active attacks; for example, it's trivial to downgrade this mechanism to plaintext by simply removing a header. Furthermore, Firefox currently doesn't check the certificate when using TLS for http:// URLs, so an attacker can pretend to be the server and the browser won't be the wiser.
In other words, if you want real security, you still need to use "full" https://.

Today I read on Patrick McManus blog that Opportunistic Encryption will be available in Firefox 37 ! (the bug report).

I have a self signed certificate, so let’s go for some testing !
I use nginx, thus I only need to modify my configuration to add the Alt-Svc header in my blog’s virtualhost.

add_header Alt-Svc 'h2=":443"; ma=60'; 
# It seems that it works too with "spdy/3.1=:443".

You can specify another host, for example h2=“secure.alteroot.org:443”. In this case, secure.alteroot.org must present a valid certificate for blog.alteroot.org (see the rfc, Clients MUST NOT use alternative services with a host that is different than the origin’s without strong server authentication.)
The “ma” is ‘max-age’, which indicates the number of seconds since the response was generated the alternative service is considered fresh for (default 24h). According to my tests, once the max-age is expired ( 60 sec without requests in this example), the browser reuses plain http request.

Of course your service must be configured on port 443 with an ssl certificate (even selfsigned).

Here a screenshot of Firefox 37 inspector.

You can see that I request http://blog.alteroot.org (first two requests), then I refresh : now my request use encryption !
You see the Alt-Svc header sent by nginx, and the client header “Alt-Used: 1” (When a client uses an alternate service, it MUST emit the Alt-Used header field on every request using that alternate service.).
And no certificate warning !

In my nginx log, I can see that the browser use a secure channel :

[28/Mar/2015:17:23:09 +0100] "GET /page2/ HTTP/1.1" 304 200 "http://blog.alteroot.org/" "Mozilla/5.0 (X11; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0" blog.alteroot.org:443 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

At the moment, only Firefox 37 support alt-svc for Opportunistic Encryption. A chromium implementation is under development.

News from Xfce 4.12, part 3

Thu, 19 Feb 2015 13:00:00 +0000

A bunch of news from the development of Xfce 4.12 !
Since my previous post in november (read it !) , many things have happened.
First, the most awaited news … wait for it… a date has been decided ! 4.12 will released by the last week end of February 2015 ! (yes, 1 week from now)

Now, I would like to draw your attention to a list of threads on the xfce4-dev mailist :

Listing our current forces : This thread has helped a lot to grant privileges on git, bugzilla and xfce internal services to more people. Now it’s easier to give access to a new contributors !
A proposition to do regular releases for translation updates
A Xfce mirror is now available on GitHub
Discussions around the 4.12 default Gtk+ theme. Maybe the ’new’ xfce-refresh from ochosi
About gtk-xfce-engine, the decision has been made to drop the gtk3 engine part completely, and leave creating themes to themers https://mail.xfce.org/pipermail/xfce4-dev/2015-February/031084.html
Participation to GSoC 2015 ?
In order to attract new contributors to Xfce, a “list of easy bugs” is now available on the wiki
4.12 String freeze will be the 20 February to let translators do their job https://mail.xfce.org/pipermail/xfce4-dev/2015-February/031127.html
New discussion about the port to GTK3. Note: some components (libxfce4ui, xfce4-panel etc) already have a partial support of gtk3.
In other news, Graeme Gott’s Whisker Menu plugin is now an official Xfce project! Andrzej’s PulseAudio plugin also joining the family!
The critical bug list has almost been eradicated, huge congrats to all developpers :)

Core Apps :

xfce4-panel :

The intelligent panel hiding feature (see my previous post) has been released in 4.11.2 !

xfwm4 :

Oh wow, many commits, such features ! You can read the 4.11.3 announcement here

Add titleless maximization feature.
Client Side Decoration (CSD) support ! (Compositor need to be enabled). A screenshot with and without on gedit (an application that need CSD)
Windows preview in alt+tab (Compositor need to be enabled too).
Nice, right ? :)

xfce4-settings :

Release of 4.11.4
Add touchpad support with libinput (yes, already !)
Better monitor and multi monitor support
Add Gtk+ theme palettes in the appearance dialog (see screenshot)
Add previews for icon themes in appearance settings (see screenshot)

xfce4-session :

Bugfixes, bugfixes, bugfixes !
The “logind runtime detection to support suspend/hibernate” patch has been merged.
Release of 4.11.1

thunar :

Add support for the GTK 3 bookmarks file,
Bugfixes, support for binary file size units
Release of 1.6.4

gtk-xfce-engine :

Drop of gtk3 part of the engine + fixes
Release of 3.1.0

xfce4-dev-tools :

Fixes, dependencies update etc
Release of 4.11.1

xfconf :

Not many exciting changes for this release, mostly translation updates, small fixes and code tweaks.. 4.11.0

Apps :

xfce4-screenshooter :

Support of imgur uploads ! Yes !
Release of 1.8.2 . Thank gaston !

xfce4-taskmanager :

Big cleanup of UI, with new treeview mode, general UI/graph overhaul and lot of bugfixes)
Release of 1.1.0 with all theses changes. Thanks gaston, ochosi and Cavalier

xfce4-power-manager :

The famous “Please bring back the tray icon” bug got some attention, with a patch from Eric to re-add a tray icon, it need extended testing though
Release of 1.4.2

Panel Plugins :

Some love for panel plugins, from Harald Judt and Landry Breuil, mostly build/bugfixes and translations updates :

xfce4-systemload-plugin 1.1.3 https://mail.xfce.org/pipermail/xfce-announce/2014-November/000343.html
xfce4-netload-plugin 1.2.4 https://mail.xfce.org/pipermail/xfce-announce/2014-November/000344.html
xfce4-weather-plugin 0.8.5 https://mail.xfce.org/pipermail/xfce-announce/2014-December/000347.html
xfce4-eyes-plugin 4.4.3 https://mail.xfce.org/pipermail/xfce-announce/2014-December/000348.html
xfce4-cpufreq-plugin 1.1.1 https://mail.xfce.org/pipermail/xfce-announce/2014-December/000349.html
xfce4-verve-plugin 1.0.1 https://mail.xfce.org/pipermail/xfce-announce/2015-January/000355.html
xfce4-indicator-plugin 2.3.3 https://mail.xfce.org/pipermail/xfce-announce/2015-January/000352.html

As you can see, xfce development is more active than ever !
So be prepared to be amazed by 4.12 in a few weeks !

Sources :

The #xfce-dev IRC chan on Freenode, THE place to be.
Xfce4 dev list
Thanks to ToZ on the xfce forum

News from xfce 4.12, part 2 !

Wed, 19 Nov 2014 21:30:00 +0000

Update 19/02/2015 : A new post is available !

Following my first post about the Xfce development status 4 months ago (you can (re) read it here), let’s go for a bunch of reviews of the last commits.

A little clarifications about comments on my 1st post : I’m NOT a core xfce dev, I’m just a small contributor, who try to help as much as I can.

Note: You can see lots of screenshots about all theses news features on the xfce forum, in the What’s New in Xfce topic from ToZ.

xfce4-power-manager :

A new xfce4-power-manager release (compatible with xfce 4.10), thanks to Eric Koegell and Simon Steinbeiss
Provide a better support of systemd and upower
Brightness plugin has been merged into the battery indicator plugin so a new plugin is born, “Power Manager plugin”
Some design changes, see screenshots
The roadmap to 1.5 : https://wiki.xfce.org/design/power-manager

xfdesktop :

Add a –next option to force wallpaper changes
Add a move to trash menu option
Add an –enable-debug option added to xfdesktop-settings
Add optional blurring on shadow on text of icons, see Screenshots

xfce4-panel :

Implementation of a intelligent auto-hide functionality by Jannis Pohlmann !
Whenever the active window changes or whenever the geometry of an already active window changes, the algorithm checks whether the active window and the panel overlap. If this is the case, the panel is hidden. Otherwise, it is made or remains visible.
A quick demo (only available in git version for now) Xfce4-panel 4.12 intelligent auto-hide feature

xfce4-session :

Add logind runtime detection to support suspend/hibernate
Support for upower 0.99
Add DragonflyBSD to host check

xfce4-settings :

Reapply settings when external keyboard connects
New display settings dialog
Add support for upower 0.99

Thunar :

Default application not respected with glib >= 2.4
Check for thumbnails in the location
Add appdata file
Add pkexec policy. This way if the user of a desktop system wants to use thunar to modify files as root and has the proper credentials they can.

Apps / Plugins :

Mousepad and Parole have been ported to gtk3 !
Partial rewrite of squeeze (Archive manager)
Multiples releases/bugfixes of xfce4-netload-plugin, xfce4-weather-plugin, xfce4-equake-plugin

Development Releases of core components :

xfdesktop 4.11.8 : https://mail.xfce.org/pipermail/xfce4-dev/2014-August/030823.html
xfwm4 4.11.2 : https://mail.xfce.org/pipermail/xfce-announce/2014-July/000329.html
xfce4-settings 4.11.3 : https://mail.xfce.org/pipermail/xfce-announce/2014-August/000333.html

Maintenances/New Releases (for xfce 4.10 ) :

Xfce4-power-manager 1.4.1 : see above in this post. http://git.xfce.org/xfce/xfce4-power-manager/tree/NEWS
xfdesktop 4.10.3 : Release of a maintenance version which fix a lot of bugs. https://mail.xfce.org/pipermail/xfce4-dev/2014-November/030941.html, thanks to Eric Koegel

Others :

A side note about light-locker, a simple session-locker for lightdm, which work really great with xfce :). If you want a nice and modern locker, try it !
Debian Jessie will use xfce 4.10 (as 4.12 is not ready on 5th November, the date of Debian Freeze)
If you are an ArchLinux user, and want to test the devel version of xfce, you can use my [xfce411] repo. NO WARRANTY, support or whatever ! Use it at your own risks. Read the README ;)
Last minute : A reorganisation of the team is running, to list our current force, identify weak spots (where we urgently need new contributors), welcome new members etc. See https://mail.xfce.org/pipermail/xfce4-dev/2014-November/030946.html for more informations !

Sources :

Thanks to ToZ on the xfce forum
The #xfce-dev IRC chan on Freenode, the best way to be involved in Xfce development
Xfce4 dev list

Slides du Premier Meetup Archlinux France

Wed, 22 Oct 2014 00:00:00 +0000

Ce 21 Octobre a eu lieu le premier meetup Archlinux France, dans les locaux d’Arkena à Paris.

Deux sujets étaient à l’ordre du jour :

Présentation de la communauté Archlinux.fr ( par moi même )
Gestion de serveurs sous Archlinux ( par seblu )

35 personnes étaient présentes, ce qui est pas mal pour une première sans trop de communication (on commence doucement !)

Les discussions ont été très intéressantes, avec la présence de plusieurs developpeurs Archlinux, entre autres Guillaume , Remy , et bien sur Seblu (qui a organisé tout ça !)

Voici mes slides, présentant Archlinux.fr, son histoire, ses projets, ses besoins. Vous pouvez d’ailleurs aussi relire ce post d’il y a quelques mois).
Seblu devrait partager les siens prochainement :) :

Merci à tous les présents, et à bientôt pour la prochaine session !

Nginx - create a cache based on upstream response time

Wed, 01 Oct 2014 00:00:00 +0000

The problem : be able to cache a backend response if it took more than 5 seconds. If not, don’t cache it!
A good challenge from @florentsolt. He solves the problem with a nodejs reverse proxy behind nginx, but I prefer a pure nginx implementation ;)

I needed that kind of configuration as a temporary workaround due to a proprietary backend which responds slowly to certain request, and quickly to anothers.
So instead of let the client wait 10 seconds or more, it’s better to render a cached response for next clients for 60 seconds, even if it’s not the more up to date data.

Finding no informations on the subject, here’s my solution (maybe not the better, but it worked for my needs). If you have a better solution (even with another software), you can add a comment :).

I use the “map” directive to set a variable and pass it via the X-Accel-Expires header (from man page : Parameters of caching can also be set directly in the response header. This has higher priority than setting of caching time using the directive. The “X-Accel-Expires” header field sets caching time of a response in seconds. ).

I use two vhosts for this trick, because we need to calculate the response time before send the response to the client.

proxy_cache_path /home/tmpcache/mycache levels=1:2 keys_zone=example:10m;

# If request_time > 3sec -> set a 60sec cache
map $request_time $longtime {
	"~^([3-9]|[1-9][0-9]+)\." 60;
	default 0;    
}

server {
	listen 127.0.0.1:8888;
	root /var/www/;
	server_name example.com; 
	location / { 
		proxy_redirect          off;
		proxy_set_header        Host $host;
		proxy_pass              http://mybackend;
		add_header X-Accel-Expires $longtime;
	}
}

server {
	listen 8080;
	root /var/www/;
	server_name example.com; 
	location / { 
		proxy_redirect          off;
		proxy_set_header        Host $host;
		proxy_pass              http://127.0.0.1:8888;
		proxy_cache             example;
		proxy_cache_valid       200 0m;
		add_header X-Cached $upstream_cache_status;
		add_header X-Time $request_time;
	}
}

A request to http://example.com:8080 will now force nginx to cache the response if it take more than 3 seconds via X-Accel-Expires: 60.
If request take less than 3 sec, a X-Accel-Expires: 0 is sent, which totally disable caching.

We can see it with curl :

# curl -I "http://example.com:8080/mylongrequest" 
HTTP/1.1 200 OK
Server: nginx/1.6.0
Date: Tue, 30 Sep 2014 16:43:35 GMT
Content-Type: text/xml
Connection: keep-alive
X-Cached: MISS
X-Time: 11.862

It took 11.8 sec ! On the next call :

# curl -I "http://example.com:8080/mylongrequest" 
HTTP/1.1 200 OK
Server: nginx/1.6.0
Date: Tue, 30 Sep 2014 16:45:05 GMT
Content-Type: text/xml
Connection: keep-alive
X-Cached: HIT
X-Time: 0.000

We hit the cache \o/

Remote code execution through bash CVE-2014-6271

Wed, 24 Sep 2014 00:00:00 +0000

A new vulnerability has been discovered in bash, related to how environment variables are processed. You can read more on redhat security blog. This vulnerability can be used to execute remote code, even via ssh, if the remote shell is bash.

In ssh, you can use the ForceCommand in sshd_config, or the “command’ option in .ssh/authorized_keys. This option ‘Specifies that the command is executed whenever this key is used for authentication. The command supplied by the user (if any) is ignored.’ (man sshd).
So you can use it for example to auto launch a backup script when the user connect, and nothing more.

For example in test/.ssh/authorized_keys,

command="/usr/local/bin/backup.sh" ssh-rsa [...thepublickey...]

Now, a “ssh test@myserver.com” will automatically launch this script, and that’s all.

With CVE-2014-6271, the flaw is triggered when extra code is added to the end of these function definitions (inside the environment variable).

$ ssh test@myserver.com '() { ignored; }; /bin/date'
Wed Sep 24 18:32:41 CEST 2014

So, instead of execute your backup script, your custom command is launched !
From there; everything is possible…
So especially update your bash package if you have this kind of configuration !

Note: of course, in this case, the remote attacker need a valid account on the machine. But if you use command/ForceCommand to provide git-shell access for example, this can be a real problem !

Sources :

http://seclists.org/oss-sec/2014/q3/649
http://bugzilla.novell.com/show_bug.cgi?id=896776
https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/
http://www.csoonline.com/article/2687265/application-security/remote-exploit-in-bash-cve-2014-6271.html

News from Xfce !

Sat, 26 Jul 2014 00:00:00 +0000

Update 19/11/2014 : A new post is available !

Some news from Xfce, my favourite Desktop Environment, that I use since something like 2006.

The development is relatively slow (the last stable version, 4.10 was released in April 2012). There is not so many developers, 1 or 2 “core” devs, and less than 10 contributors (who are generally distributions maintainers, from debian, xubuntu, gentoo, arch, thanks to them !).

There was a roadmap for 4.12, where it was planned to release 4.12 mid-2013. But, hey, it’s open source, it will be out when “it will be ready” :-).

Some weeks ago, it was decided to establish a list of “critical bugs” to be eradicated in order to release xfce 4.12.
You can find the list here : https://wiki.xfce.org/releng/4.12/roadmap/critical-bugs.
Xfce 4.12 will still use gtk2, with some support of gtk3 for better integration.
Port to gtk3 will maybe be done for the next version.

What will be new in xfce 4.12 ?

All major components are already available in development version (4.11), here are a small list of what to expect :

xfwm4 :

Zoom mode. I wrote about this cool new feature in december
New, themable tabwin (alt+tab)

Changelog at http://git.xfce.org/xfce/xfwm4/tree/NEWS

xfce4-settings :

The display settings have been much improved with supporting extended desktop mode for 2 monitors and an improve minimal dialog.
Changelog at http://git.xfce.org/xfce/xfce4-settings/tree/NEWS

xfdesktop :

Lot of works on xfdesktop, already 6 pre releases!
Per-workspace wallpapers support (finally !)
Better multi monitor handling
Changelog at http://git.xfce.org/xfce/xfdesktop/tree/NEWS

xfce4-panel :

Support of gtk3 plugins
Better behavior of buttons/menus
Changelog at http://git.xfce.org/xfce/xfce4-panel/tree/NEWS

And lot of works on other components, like xfce4-power-manager (systemd support), xfburn, xfce4-mixer etc There is still some works/tests to be done on upower or systemd support for example.

Update : You can see lot of screenshots of new features on the Xfce forum, by ToZ : https://forum.xfce.org/viewtopic.php?id=8945

Buildbot

A new buildbot based on jenkins is available since few days on http://buildbot.xfce.org

Bountysource

It’s easier to copy the mail from Simon Steinbeiß to explain this part :

To get to the point: we see bountysource[2] as an easy way to offer
the community with a way to financially support Xfce. There are two
avenues a backer can choose from.
1) Set a bounty on a specific bug (we've pulled in all the reports for
many components already, so you can easily find them on
bountysource.com)
2) Back the Xfce team

Update: More explanations about bountysource: https://mail.xfce.org/pipermail/xfce4-dev/2014-July/030807.html

So if you are interested yo help Xfce, go to the contribute wiki page !

Chiffrement entre serveurs avec starttls et postfix

Wed, 02 Jul 2014 00:00:00 +0000

Depuis quelques temps, la question du chiffrement des mails en transit (de serveurs à serveurs) revient d’actualité. On peut voir google qui diffuse des statistiques dans son report “Safer email”,

En effet dans l’état actuel, si vous avez suivi le setup d’un serveur mail, les connexions sont chiffrées entre le client (thunderbird par exemple) et votre serveur mail, mais pas lors de l’envoi d’un mail vers un domaine distant (de serveur à serveur donc).

Par exemple, quand j’envoi un mail vers example.com depuis mondomaine.org :

client mail ===> smtp.mondomaine.org ---> smtp.example.com ===> client mail

===> : chiffré
---> : non chiffré

On voit bien que le mail transite en clair sur l’internet à un moment. Dommage de chiffrer aux deux bouts si le transit sur le net se fait en clair !

Pour remédier à cela, c’est assez simple avec postfix (et encore plus avec opensmtpd), il suffit de modifier quelques options dans la configuration de postfix pour que starttls soit actif, fichier /etc/postfix/main.cf :

smtpd_tls_security_level = may # Cette directive remplace smtpd_use_tls=yes
smtp_tls_security_level = may # "may" pour autoriser l'envoi de mail si le serveur distant ne supporte pas STARTTLS (ce qui est encore relativement courant).
smtpd_tls_exclude_ciphers = aNULL, MD5, DES, ADH # On desactive certains ciphers pour des raisons de sécurité...
smtpd_tls_loglevel = 1 # Et on ajoute quelques logs pour voir si tout se passe bien 
smtp_tls_loglevel = 1

Il est possible d’être bien plus stricte sur la directive smtp_tls_security_level, voir http://www.postfix.org/postconf.5.html#smtp_tls_security_level, à vous de tester selon vos besoins.

Vous pouvez ensuite tester votre configuration sur https://starttls.info/, ou via sslyze (merci Puckel pour l’info) :

Note : Postfix supporte DANE depuis la version 2.11, qui permet une validation des certificats via DNSSEC. Ayant du DNSSEC sur alteroot.org, cela sera surement l’objet d’un prochain article :)

Sources :

http://www.postfix.org/postconf.5.html#smtpd_tls_exclude_ciphers
http://www.postfix.org/postconf.5.html#smtp_tls_security_level

Fail2ban on FreeBSD

Sat, 14 Jun 2014 00:00:00 +0000

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits…
It can be usefull to ban bots who try to bruteforce your ssh and flood your logs (another solution is to restrict allowed IP or change sshd port).

Here a quick how-to to enable fail2ban on FreeBSD, in combination with pf (packet filter).

Update 07/05/2015 : This works on FreeBSD 9.x. I haven’t tested it against FreeBSD 10.x yet.

First, install it, use pkg, portinstall, whatever :

# pkg install py27-fail2ban

Then go to /usr/local/etc/fail2ban and create the file jail.d/ssh-pf.local

[ssh-pf]
enabled  = true
filter   = sshd
action   = pf
#          sendmail-whois[name=SSH, dest=root@localhost, sender=noreply@localhost]
logpath  = /var/log/auth.log
findtime  = 600
maxretry = 3
bantime  = 3600

You can of course configure maxretry/bantime/findtime or sending mails.
I use pf in action (‘action = pf’), you can see what this action do in action.d/pf.conf

[Definition]
actionstart = 
actionstop = 
actioncheck = 
actionban = /sbin/pfctl -t <tablename> -T add <ip>/32
actionunban = /sbin/pfctl -t <tablename> -T delete <ip>/32
[Init]
tablename = fail2ban

When ‘action’ is triggered, fail2ban launch pfctl -t <tablename> -T add <ip>/32 to add it to pf table ‘fail2ban’.
It’s now ok on fail2ban side (easy right ?)

On pf side, in /etc/pf.conf (blank by default, adapt it if you already have pf rules), you need to add a rule to block all IPs in the fail2ban table, for example :

ext_if="re0" # your interface ! 
table <fail2ban> persist
block quick proto tcp from <fail2ban> to $ext_if port ssh

Now, we need to start pf, then fail2ban :

/etc/rc.d/pf onestart
/usr/local/etc/rc.d/fail2ban onestart

I use onestart, with the assumption that it’s the first start of pf and fail2ban, you will need to add pf and fail2ban in /etc/rc.conf to auto start them on boot.

Now you can look in /var/log/fail2ban.log to see detected IP and applied ban.

To list current banned IP :

pfctl -t fail2ban -T show

Sources:

Archlinux.fr passé, présent, futur

Sun, 18 May 2014 00:00:00 +0000

Archlinux.fr est le principal site francophone autour de la distribution Archlinux. Comme tout site animé par une communauté, il a subi des hauts et des bas au niveau de son activité. Je vais tenter de faire un état des lieux et des projets futurs.

D’abord, un petit “About me”, je suis dans l’aventure archlinux.fr depuis le quasi début (septembre 2006), et admin depuis cette époque. Je me suis co-occupé de la partie serveurs/miroirs toutes ces années (c’est mon métier après tout), et me retrouve à l’heure actuelle le dernier admin actif. D’ou ce blogpost :)

Archlinux.fr existe depuis aout 2006, suite à une scission d’ avec archlinuxfr.org (maintenant disparu) après plusieurs désaccords, un fork comme on les connait dans le monde du libre (je suis arrivé à ce moment là).

Archlinux.fr, c’est

Un portail, qui traduit généralement les news en français, et menant vers les autres ressources
Un forum d’entraide en français, ayant plusieurs milliers d’inscrits (pas tous actifs ;))
Un wiki
Un chan IRC #archlinux-fr sur freenode
Un miroir officiel
Un miroir fournissant des delta
AFUR, un “AUR” à la française, ou vous pouvez proposer vos paquets précompilés, utilisables via archlinuxfr
Un planète regroupant quelques blogs français
Et l’un des softs les plus connus d’archlinux : yaourt !

Le tout tourne sur 2 serveurs dédiés + un mutualisé, évidemment les dédiés tournent sur archlinux, avec un kernel linux-lts ;)

Un pour le site/wiki/forum
Un pour la partie miroir/afur
Le mutu pour les mails/dns/historiquement :)

Les projets en cours / à venir

Léger redesign du site/wiki (resynchronisation avec le design du .org)
A venir un redesign du forum ! (si Mike se motive)
Mise en place du support des signature de paquets sur [archlinuxfr] (disponible depuis pacman 4)
Trouver un vrai mainteneur pour yaourt (j’y reviens plus loin)
Tout passer en https par défaut (déjà disponible, mais pas forcé, merci globalsign pour le certificat !)
Pourquoi pas ajouter un miroir archlinuxarm

Yaourt, la bonne crème

Yaourt est une surcouche à Pacman, le gestionnaire de paquets d’archlinux. C’est l’une des plus vieilles surcouches encore en activité, et je pense la plus utilisée (enfin, moins depuis 2/3 ans). Écrit au départ par Wain (un des fondateurs de archlinux.fr), entièrement en bash, il a ensuite été réécrit et splitté en 2 parties par notre ami Tuxce.

package-query en C, pour toute la partie alpm / aur
yaourt en bash pour l’interface utilisateur

Package-query a grandement aidé à améliorer la vitesse d’exécution de yaourt.
Malheureusement depuis environ 1-2 ans, yaourt évolue très peu, par faute de mainteneur (wain et tuxce ayant d’autres priorités personnelles). Son code source est disponible sur github, il y a plusieurs bugs ouverts et des pull request en cours. Il y a donc besoin de bras pour cette partie ! Donc si vous vous sentez l’âme d’un mangeur de yaourt, la porte est ouverte :) Vous pouvez me contacter soit dans les commentaires, soit sur IRC, soit par mail, twitter etc pour en discuter.
J’ écrirai une version anglais à ce propos afin de toucher un plus large public :)

Et pour finir un merci à tous ceux qui participent/ont participés à archlinux.fr !

Replace a dead node in Cassandra

Wed, 12 Mar 2014 00:00:00 +0000

Note (June 2020): this article is old and not really revelant anymore. If you use a modern version of cassandra, look at -Dcassandra.replace_address_first_boot option !

I want to share some tips about my experimentations with Cassandra (version 2.0.x).

I found some documentations on datastax website about replacing a dead node, but it is not suitable for our needs, because in case of hardware crash, we will set up a new node with exactly the same IP (replace “in place”). Update : the documentation in now up to date on datastax !

If you try to start the new node with the same IP, cassandra doesn’t start with :

java.lang.RuntimeException: A node with address /10.20.10.2 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.

So, we need to use the “cassandra.replace_address” directive (which is not really documented ? :() See this commit and this bug report, available since 1.2.11/2.0.0, it’s an easier solution and it works.

+    - New replace_address to supplant the (now removed) replace_token and
+      replace_node workflows to replace a dead node in place.  Works like the
+      old options, but takes the IP address of the node to be replaced.

It’s a JVM directive, so we can add it at the end of /etc/cassandra/cassandra-env.sh (debian package), for example:

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.20.10.2"

Of course, 10.20.10.2 = ip of your dead/new node.

Now, start cassandra, and in logs you will see :

INFO [main] 2014-03-10 14:58:17,804 StorageService.java (line 941) JOINING: schema complete, ready to bootstrap
INFO [main] 2014-03-10 14:58:17,805 StorageService.java (line 941) JOINING: waiting for pending range calculation
INFO [main] 2014-03-10 14:58:17,805 StorageService.java (line 941) JOINING: calculation complete, ready to bootstrap
INFO [main] 2014-03-10 14:58:17,805 StorageService.java (line 941) JOINING: Replacing a node with token(s): [...]
[...]
INFO [main] 2014-03-10 14:58:17,844 StorageService.java (line 941) JOINING: Starting to bootstrap...
INFO [main] 2014-03-10 14:58:18,551 StreamResultFuture.java (line 82) [Stream #effef960-6efe-11e3-9a75-3f94ec5476e9] Executing streaming plan for Bootstrap

Node is in boostraping mode and will retrieve data from cluster. This may take lots of time.
If the node is a seed node, a warning will indicate that the node did not auto bootstrap. This is normal, you need to run a nodetool repair on the node.

On the new node :

# nodetools netstats

Mode: JOINING
Bootstrap effef960-6efe-11e3-9a75-3f94ec5476e9
    /10.20.10.1
	Receiving 102 files, 17467071157 bytes total
[...]

After some time, you will see some informations on logs !
On the new node :

 INFO [STREAM-IN-/10.20.10.1] 2014-03-10 15:15:40,363 StreamResultFuture.java (line 215) [Stream #effef960-6efe-11e3-9a75-3f94ec5476e9] All sessions completed
 INFO [main] 2014-03-10 15:15:40,366 StorageService.java (line 970) Bootstrap completed! for the tokens [...]
[...]
 INFO [main] 2014-03-10 15:15:40,412 StorageService.java (line 1371) Node /10.20.10.2 state jump to normal
 WARN [main] 2014-03-10 15:15:40,413 StorageService.java (line 1378) Not updating token metadata for /10.20.30.51 because I am replacing it
 INFO [main] 2014-03-10 15:15:40,419 StorageService.java (line 821) Startup completed! Now serving reads.

And on other nodes :

 INFO [GossipStage:1] 2014-03-10 15:15:40,625 StorageService.java (line 1371) Node /10.20.10.2 state jump to normal

Et voilà, dead node has been replaced !
Don’t forget to REMOVE modifications on cassandra-env.sh after the complete bootstrap !

Enjoy !

Mise en place de DKIM avec Postfix

Thu, 30 Jan 2014 00:00:00 +0000

Cette documentation décrit la mise en place de DKIM (DomainKeys Identified Mail), une norme d’authentification du nom de domaine de l’expéditeur d’un mail.

Elle fait suite à la documentation de mise en place d’un serveur mail.

On prendra en domaine de test domain.org et comme nom de serveur server.domain.org

On utilise opendkim pour gérer dkim. Postfix transmet ensuite le mail à OpenDKIM (via socket).

# apt-get install opendkim opendkim-tools
# mkdir -p /etc/opendkim

On génère les clés, et on utilise “server” en tant que selector (choisi arbitrairement, on peut aussi utiliser “default”).

# opendkim-genkey -D /etc/opendkim/ -d domain.org -s server
# rm /etc/opendkim.conf
# vim /etc/opendkim/opendkim.conf

UserID    opendkim:opendkim
Domain    domain.org
KeyFile  /etc/opendkim/server.private
Selector        server
AutoRestart     yes
DNSTimeout  5
UMask      002

# ln -s /etc/opendkim/opendkim.conf /etc/opendkim.conf
# chown opendkim:opendkim -R /etc/opendkim
# vim /etc/default/opendkim
 
SOCKET="local:/var/spool/postfix/var/run/opendkim/opendkim.sock"
 
# mkdir /var/spool/postfix/var/run/opendkim
# chmod 775 /var/spool/postfix/var/run/opendkim
# chown opendkim: /var/spool/postfix/var/run/opendkim
# adduser postfix opendkim
# vim  /etc/postfix/main.cf

# Ce chemin est en réalité /var/spool/postfix/var/run/opendkim/opendkim.sock 
# car nous sommes dans le chroot de postfix 
smtpd_milters           = unix:/var/run/opendkim/opendkim.sock
non_smtpd_milters       = $smtpd_milters
milter_default_action   = accept

Copiez les lignes de /etc/opendkim/server.txt dans votre zone DNS.

# /etc/init.d/opendkim restart
# /etc/init.d/postfix restart

Vous pouvez tester la bonne configuration sur http://www.appmaildev.com/en/domainkeys/ ou via dig :

# "server" est notre selector
# dig TXT server._domainkey.domain.org

Liens :

http://www.opendkim.org/

http://www.postfix.org/MILTER_README.html

Xfce - Xfwm4 zoom mode in 4.12

Wed, 11 Dec 2013 00:00:00 +0000

Xfce is my main desktop environment since more than 6 years, and I really like it. I try to make some patch from time to time, and if you search a project to contribute, it’s here ;-)

The core team is really small (2 or 3 people), so development evolves rather slowly, and the 4.12 will be released when it is ready.

One of the latest feature is the implementation of a compositor zoom, like the compiz ezoom plugin, from an external developper ( see this thread).

Here a little video with latest git version ( from http://git.xfce.org/xfce/xfwm4/ ). You just need to press ALT and scroll up/down to zoom in/out.

Xfwm4 4.12 zoom feature

(yeah, it’s an excuse to test the html5 <video> balise with ogv file ;)).

I will try to make some blog posts about new features in xfce world for the upcoming 4.12 (in 2014 I hope!), stay tuned !

Use KDEConnect with android and Xfce

Sat, 16 Nov 2013 00:00:00 +0000

Update 14/06/2014 : kdeconnect is in community + typo

KdeConnect allow you to share notifications (sms, phone calls), files, clipboard and more things between your Android and your Linux desktop.

It’s designed to work with KDE, but you can use it on other desktop environment (like xfce!), modulo some dependencies installations. You can look at Albert Vaka’s blog for more informations and demonstrations.

I use ArchLinux, and Kdeconnect is available ~~on AUR~~ in community, here are the steps to run kdeconnect on Xfce, with a fewer dependencies (but you’ll still need kdelibs).

$ yaourt -Sy kdeconnect
[...]
# No need to launch qdbus command, this will not work yet
# Some other dependencies to allow qdbus to works...
$ yaourt -S qt5-tools
# And to manager notifications, you need knotify4 (who will give a hand to your notify daemon, xfce4-notifyd for me)
$ yaourt -S kdebase-runtime

Ok, so now, kdeconnect is installed. You need to relaunch Xfce with KDE compatibility enabled (to launch kdeinit on startup).

To do that, you can go to xfce4-settings-manager -> Session and startup -> advanced and enable ‘Launch KDE services on startup’, or use xfconf-query :

$ xfconf-query -c xfce4-session -p /compat/LaunchKDE -s true

Restart your session ! Now, open a terminal and launch the qdbus command :

$ qdbus org.kde.kded /kded loadModule kdeconnect 
true
$ kbuildsycoca4 -noincremental
[...]

Note: you’ll need to autolaunch “qdbus org.kde.kded /kded loadModule kdeconnect” on each startup.

Install the Android application on your smartphone, and you are ready !

The application should detect your kdeconnect setup (you MUST be on the same wifi network).

You can now ping your laptop and a notification should appear on you desktop ! You can share you clipboard, pictures, sms text etc…

On the kdeconnect android application, you can enable/disable features (like the shared clipboard which can be hassle).

For some debug look at you ~/.xsession-errors file.

Here are some screenshots :

Playing with the new ownCloud share API

Sat, 02 Nov 2013 00:00:00 +0000

I use OwnCloud since v2, and really enjoy it, mainly since desktop clients are available. I can easily sync a file if I copy it in my ~/ownCloud/ directory (dropbox style) !

But one of the missing feature for me is a share API. You can synchronize files from your desktop client or owncloud smartphone apps, but you CAN’T share a file without going to the web interface… Pretty annoying. I need this feature to quickly share a file with a public URL (share on IRC or IM)

Since owncloud v6 ( ~~still in beta stage ATM~~ Now released !), a share api is now available!

The documentation is available here on the owncloud github account. So let’s try ! (Note: it only works on owncloud 6 !)

It’s a REST API, so you can use it simply via cURL call. Here are some examples.

Get all shares for your user ( you need your session cookie to be authenticated during requests):

Request :

curl "https://your.instance.com/ocs/v1.php/apps/files_sharing/api/v1/shares" -k -H 'Cookie: ocxxxxx=xxxxxxxxx'

Response :

<?xml version="1.0"?>
<ocs>
 <meta>
  <status>ok</status>
  <statuscode>100</statuscode>
  <message/>
 </meta>
 <data>
  <element>
   <id>1</id>
   <item_type>file</item_type>
   <parent/>
   <share_type>3</share_type>
   <share_with/>
   <file_source>6</file_source>
   <path>/ownCloudUserManual.pdf</path>
   <permissions>1</permissions>
   <stime>1383432712</stime>
   <expiration/>
   <token>yyyyyyyyyyyyyyyyyyyyyy</token>
   <storage>1</storage>
   <mail_send>0</mail_send>
  </element>
  <element>
   <id>2</id>
   <item_type>folder</item_type>
   <parent/>
   <share_type>3</share_type>
   <share_with/>
   <file_source>5</file_source>
   <path>/photos</path>
   <permissions>1</permissions>
   <stime>1383434668</stime>
   <expiration/>
   <token>31ca68aae4d357cf6c89598983cd0c25</token>
   <storage>1</storage>
   <mail_send>0</mail_send>
  </element>
 </data>
</ocs>

Create a public share (type 3) for “Pictures/picture1.png”

Request :

curl "https://your.instance.com/ocs/v1.php/apps/files_sharing/api/v1/shares" -k -u $USER:$PASS -X POST --data "path=Pictures/picture1.png&shareType=3"

Response :

<?xml version="1.0"?>
<ocs>
 <meta>
  <status>ok</status>
  <statuscode>100</statuscode>
  <message/>
 </meta>
 <data>
  <id>1</id>
  <url>https://your.instance.com/public.php?service=files&amp;t=xxxxxxxxxxxxxxxxxxxxxxxxxxx</url>
  <token>yyyyyyyyyyyyyyyy</token>
 </data>
</ocs>

File is now available publicy to https://your.instance.com/public.php?service=files&t=xxxxxxxxxxxxxxxxxxxxxxxxxxx !

So, you can use a simple shell script to share a file automagically and copy the public URL to your clipboard. You need bash, xclip and xpath (perl-xml-xpath -ArchLinux- or libxml-xpath-perl -Debian-).

Update 2014/03/01 : A public link creator script is now available on http://blog.schiessle.org/2013/12/30/the-owncloud-public-link-creator/ :)

#!/bin/bash
# This is a quick & dirty shell script, consider it as a proof of concept ;) 
# License: WTFPL v2 
# Author: Skunnyk{@}alteroot.org
# Change theses 3 values
url="https://your.instance.com"
user="User"
pass="Password"
# You should not touch theses values
curlopts="-s -k"
apiurl="ocs/v1.php/apps/files_sharing/api/v1/shares"

# Some checks
for executables in xpath curl xclip; do
    command -v $executables  >/dev/null 2>&1 || { echo >&2 "I need $executables but it's not installed."; exit 1; }
done

usage() {
    echo ""
    echo ""
    echo "USAGE : "
    echo "$(basename $0) path/to/file"
    echo "Don't forget to change url/user/pass variables in this script"
    echo ""
}

if [[ $# -ne 1 ]]; then
    usage
    exit 1
fi

file="$1"

command="curl $curlopts $url/$apiurl -u $user:$pass -X POST --data path=$1&shareType=3"

result=$($command | xpath "/ocs/data/url/text()" 2>/dev/null)
if [ -z $result ]; then
    echo "Error on share..." 
    exit 1
else
    echo $result | sed "s/amp;//" | xclip
    xclip -o
fi

Now, you juste need to launch “share-me.sh MyDocument” or “share-me.sh photos/photo.png”, and the public share URL will be copied to your clipboard.

I hope this feature will be added soon in the desktop and android/ios client :)

Enjoy,

Mise en place d'un serveur mail - Postfix, Dspam et Dovecot

Mon, 07 Oct 2013 00:00:00 +0000

Update 14/01/14 : ajout de la liste complète des paquets nécessaires et correction de quelques typos. Thx @Puckel_ !

Cette documentation décrit la mise en place de postfix, dspam et dovecot pour un usage simple, géré avec des utilisateurs et domaines virtuels. Le stockage se fait en fichiers, que ce soit pour les règles antispam ou les mails (pas besoin de databases).

Le tout en mode sécurisé, avec de l’imap et du smtp via starttls. Il est aussi possible de signaler un spam simplement en le déplaçant dans le dossier “SPAM” afin que dspam l’apprenne.

Les doc existantes ne me convenaient pas, soit obsolètes, soit demandant de desactiver le chroot de postfix, soit trop de hacks dans le master.cf, ou bien incomplètes…

On prendra en domaine de test domain.org et comme nom de serveur server.domain.org

Le fonctionnement est le suivant :

Postfix -> dspam -> dovecot

Postfix transmet le mail à dspam (via socket), qui analyse le mail, puis le retourne (toujours via socket) à dovecot qui le stocke.

Dovecot centralise l’authentification via sasl, que ce soit pour imap ou pour smtp.

On utilise les version suivante (debian wheezy):

postfix 2.9 : smtp
dspam 3.10  : antispam 
dovecot 2.1 : imap

On a aussi besoin des paquets suivants :

postfix
dspam
libdspam7-drv-hash
dovecot-antispam
dovecot-managesieved
dovecot-sieve
dovecot-imapd
dovecot-lmtp

Tout d’abord, on créé un user vmail, qui s’occupera du stockage des mails (ici /home/mail)

addgroup --gid 5000 vmail
adduser --home /home/mail/ --uid 5000 --gid 5000 --shell /bin/false vmail

On configure Postfix, les commentaires sont integrés :

smtpd_tls_cert_file=/etc/ssl/domain.chain.crt
smtpd_tls_key_file=/etc/ssl/domain.key
smtpd_use_tls=yes

myhostname = server.domain.org
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = server.domain.org, localhost.localdomain, localhost

virtual_mailbox_domains = domain.org # completer avec d'autres domaines, séparé par des virgules
virtual_mailbox_base = /home/mail/vhosts # ou sont physiquement stockés les mails
virtual_mailbox_maps = hash:/etc/postfix/virtual_mailbox_recipients # le mapping entre l'adresse et le stockage physique 
virtual_uid_maps = static:5000 # tout est géré via l'utilisateur vmail, qui a l'id 5000
virtual_gid_maps = static:5000
virtual_alias_maps = hash:/etc/postfix/virtual_alias_recipients # en cas d'alias (par exemple root@ -> posmaster@)
smtpd_sasl_auth_enable = yes # activer l'auth sasl pour l'envoi de mail
smtpd_sasl_local_domain =
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot # On passe par dovecot pour l'auth sasl
smtpd_sasl_path = private/auth # la socket d'authentifiction
broken_sasl_auth_clients = yes # pour allow de vieux clients mails
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, reject_unverified_recipient
smtpd_tls_auth_only = yes
virtual_transport  = lmtp:unix:dspam/dspam.sock # et on transmet le mail à dspam

Les fichiers virtual_mailbox_recipients et virtual_alias_recipients sont de la forme suivante :

$ cat virtual_mailbox_recipients 
myuser@domain.org    domain.org/myuser

domain.org/myuser correspond au stockage physique, ce qui va se transformer en /home/mail/vhosts/domain.org/myuser

$ cat virtual_alias_recipients
aliasuser@domain.org myuser@domain.org

Il faut ensuite générer les fichiers .db pour que postfix puisse les utiliser, à faire à chaque modification

$ postmap virtual_mailbox_recipients
$ postmap virtual_alias_recipients

Reste à modifier le master.cf de postfix afin d’activer le port submission (port 587) pour l’envoi de mails. Décommenter les lignes suivantes :

submission inet n       -       -       -       -       smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o milter_macro_daemon_name=ORIGINATING

On est bon coté postfix.

Attaquons dspam. On a vu que postfix passe le mail à dspam via dspam/dspam.sock, qui se situe dans le chroot de postfix, par défaut dans /var/spool/postfix/ . On créé donc un dossier /var/spool/postfix/dspam :

mkdir /var/spool/postfix/dspam
chown dspam: /var/spool/postfix/dspam

Editer le /etc/default/dspam

START=yes

Editer le /etc/dspam/dspam.conf

DeliveryHost            /var/run/dovecot/lmtp # On delivera le mail à dovecot après le parsing antispam
DeliveryIdent           dspam.domain
DeliveryProto           LMTP

Trust vmail # C'est l'utilisateur vmail qui déplace dans le dossier SPAM via les règles sieve

Preference "tagSpam=on" # On veut que les spams soient taggués  

ServerMode auto
ServerParameters        "--deliver=innocent,spam" # On veut que dspam délivre aussi les mails taggués en spam (que l'ont mettra dans le dossier SPAM via sieve)
ServerDomainSocketPath  "/var/spool/postfix/dspam/dspam.sock" # Le path dans le chroot de postfix

Ensuite dans /etc/dspam/default.prefs

spamAction=deliver # On veut delivrer dans la mailbox, pas en quarantaine

On passe à dovecot. Debian split la configuration dans de multiples fichier sous /etc/dovecot/conf.d/, ici une version simplifiée, à adapter à la configuration existante (il vous faudra modifier 10-auth.conf 10-mail.conf 10-ssl.conf 15-lda.conf 20-imap.conf 10-director.conf 10-master.conf 20-lmtp.conf 20-managesieve.conf 90-plugin.conf 90-sieve.conf auth-passwdfile.conf.ext):

disable_plaintext_auth = yes
auth_mechanisms = plain login
!include auth-passwdfile.conf.ext # On y revient plus tard

mail_location = maildir:/home/mail/vhosts/%d/%n # %d = domaine %n = user
namespace inbox {
separator = /
inbox = yes
}
mail_uid = 5000 # Utilisateur vmail
mail_gid = 5000

service lmtp {
unix_listener lmtp {
user = vmail # On tourne avec le user vmail
}
}

service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
user = postfix
group = postfix
}
}

ssl = required
ssl_cert = </etc/ssl/domain.chain.crt
ssl_key = </etc/ssl/domain.key

postmaster_address = root@domain.org

protocol lda {
auth_socket_path = /var/spool/postfix/private/auth
}

protocol imap {
mail_plugins = $mail_plugins antispam # On load le plugin antispam
}

protocol lmtp {
# Space separated list of plugins to load (default is global mail_plugins).
mail_plugins = $mail_plugins sieve # On load le pluin sieve pour trier les mails
}

# On active managesieve, afin de pouvoir gérer les règles sieve
service managesieve-login {
inet_listener sieve {
port = 4190
}

service managesieve {
}

protocol sieve {
}

# On passe à la configuration antispam, afin de faire apprendre à dspam lorsque vous placez manuellement dans le repertoire SPAM

plugin {
antispam_backend = dspam
antispam_dspam_binary = /usr/bin/dspam
antispam_dspam_args = --deliver;--user;%u
antispam_spam = SPAM; Spam
antispam_trash = Trash
antispam_unsure = Trash
antispam_signature = X-DSPAM-Signature
}


plugin {
# The path to the user's main active script. If ManageSieve is used, this the
# location of the symbolic link controlled by ManageSieve.
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_before = /home/mail/sieve/before/ # Cette règle sera appliquée à TOUS les mails
}

Dans le fichier auth-passwdfile.conf.ext:

passdb {
	driver = passwd-file
	args = scheme=CRYPT username_format=%u /etc/dovecot/users
}
userdb {
	args = uid=vmail gid=vmail home=/home/mail/vhosts/%d/%n allow_all_users=yes
	driver = static
}

C’en est fini pour Dovecot.

On utilise le driver static, car dans ce setup tout est géré via l’utilisateur vmail.

Pour créer un user, il faut remplir le fichier /etc/dovecot/users. Comme postfix utilise dovecot pour l’authentification, tout se passe ici, que ce soit imap ou smtp.

L’ajout d’un utilisateur se fait de la façon suivante :

echo "myuser@domain.org:$(doveadm pw -s CRYPT -p mypassword)" >> /etc/dovecot/users
echo myuser@domain.org domain.org/myuser >> /etc/postfix/virtual_mailbox_recipients
postmap /etc/postfix/virtual_mailbox_recipients

Il est ensuite simple de faire un script shell pour créer les users en une commande.

On voit qu’une règle sieve par défaut est appliquée à tous les mails. Je m’en sers pour mettre les mails taggués en SPAM dans le dossier spam automatiquement.

$ vim /home/mail/sieve/before/spam.sieve
require ["fileinto"];
# rule:[SPAM]
if anyof (header :contains "X-DSPAM-Result" "Spam")
{
    fileinto "SPAM";
    stop;
}

# Puis on compile la règle
sievec /home/mail/sieve/before/spam.sieve

J’utilise Roundcube en tant que webmail, la seul configuration à ajouter est de charger le plugin “managesieve”, afin que chaque utilisateur puisse éditer ses règles de filtrage via roundcube.

$ vim roundcube/config/main.inc.php
$rcmail_config['plugins'] = array('managesieve');

Avec un client mail, il suffit de se connecter en imap sur le port 143 (avec starttls) et le port 587 pour l’envoi (toujours en starttls).

En bonus, pour implementer DKIM, c’est par ici : http://blog.alteroot.org/articles/2014-01-30/mise-en-place-dkim-postfix.html

Sources:

Share on Wallabag from Netvibes

Mon, 30 Sep 2013 00:00:00 +0000

Update 2014/30/01 : Poche is now Wallabag ! (see the blog for more informations). A WAAS (Wallabag As A Service) is available on Framabag :)

~~Poche~~ Wallabag is an open source self hosted read-it-later app, like pocket, readability… So you can install it on your server, or use it via wallabag hosting.

With the new Share on Netvibes, you can add unlisted web services. Go to your account share options, clic on “Add a custom link”, and complete with needed parameters. For example, to share on Wallabag demo service :

Name : Wallabag
Url : http://demo.wallabag.org/index.php?plainurl=${url}
Icon Url: http://demo.wallabag.org/themes/default/img/logo.svg

Save, and go back to your dashboard. Now you can share to your Wallabag instance !

Et voilà :)

Setup de Mozilla Sync Server

Wed, 18 Sep 2013 00:00:00 +0000

Firefox sync est disponible depuis Firefox 4, et il est possible de l’installer en auto hébergement. Je m’en sers pour synchroniser firefox desktop avec celui de mon android. L’installation est relativement simple, surtout sur une Debian, il suffit de suivre la documentation de mozilla.

Mon setup est en place avec du Nginx et Gunicorn (mozilla-sync étant en python).

Évidement, le tout se fait via une connexion https, et c’est là que les ennuis commencent…

Premièrement, si c’est un certificat auto-signé, il ne faut pas oublier d’importer votre CA certificat dans votre Firefox Android ! Sinon il sera impossible pour votre smartphone de se connecter au sync server car le certificat ne sera pas reconnu …
Ensuite, si vous avez suivi la documentation, vous devriez avoir une sync qui fonctionne avec votre Firefox desktop, via une connexion sécurisée.Le problème se situe lors du setup de Sync via Firefox Android, sur l’écran “Serveur personnalisé”, vous avez beau entrer la bonne url et mots de passe, le seul résultat est “Please enter a valid server URL”, et absolument RIEN dans les logs nginx coté serveur… Même constatation via la fonction “Pair a device”.

Après une sortie de tshark afin de voir les trames réseau, il s’avère que firefox android ne sait négocier le tls qu’AVEC un cipher RC4, ce qui est confirmé par le code mozilla-central/mobile/android/base/sync/net/TLSSocketFactory.java :

public class TLSSocketFactory extends SSLSocketFactory {
  private static final String LOG_TAG = "TLSSocketFactory";
  private static final String[] DEFAULT_CIPHER_SUITES = new String[] {
    "SSL_RSA_WITH_RC4_128_SHA",        // "RC4_SHA"
  };
  private static final String[] DEFAULT_PROTOCOLS = new String[] {
    "SSLv3",
    "TLSv1"
  };

  // Guarded by `this`.
  private static String[] cipherSuites = DEFAULT_CIPHER_SUITES;

  public TLSSocketFactory(SSLContext sslContext) {
    super(sslContext);
  }

Le problème est que les versions récente de nginx utilisent par défaut “ssl_ciphers HIGH:!aNULL:!MD5;”, et SSL_RSA_WITH_RC4_128_SHA n’est pas considéré comme un cipher “high”, donc la négociation ssl échoue tout bêtement…

Il faut donc forcer nginx à utiliser RC4, ce qui est dommage au vu de http://blog.ivanristic.com/2013/09/updated-best-practices-deprecate-rc4.html.

De plus, il semble que le client Android ne supporte pas le SNI, ce qui est dommage car vous devez donc faire la modification de ciphers dans le vhost ssl par défaut.

Voici donc une configuration fonctionnelle :

server {
	listen 443 ssl default;
	server_name mozillasync.domain.org;
	ssl_certificate /etc/nginx/ssl/cert.chain.crt;
	ssl_certificate_key /etc/nginx/ssl/cert.key;
	access_log  /var/log/nginx/mozillasync.access.log;
	ssl_ciphers RC4:HIGH:!aNULL:!MD5;
	ssl_prefer_server_ciphers on;

	location /sync/ {
		rewrite  ^/mozillasync/(.+)$ /$1 break;
		proxy_pass_header Server;
		proxy_set_header Host $http_host;
		proxy_redirect off;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Scheme $scheme;
		proxy_connect_timeout 10;
		proxy_read_timeout 10;
		proxy_pass http://localhost:5000/;
		}
	location / {
		root /home/server-full/;
	}
}

Et un lien vers le rapport de bug chez mozilla : https://bugzilla.mozilla.org/show_bug.cgi?id=756763#c31

Hotot twitter client on OpenBSD

Mon, 09 Sep 2013 00:00:00 +0000

I use Hotot on my linux setup everyday, it’s the best twitter client on Linux for me, even if it is not well maintained (maybe polly will surpass hotot a day…)

But Hotot is not available on OpenBSD ! (yes, I use it too, on -current). A port existed on openbsd-wip github repository, but since it submission on ports@openbsd.org, it diseappeared from openbsd-wip and never been accepted on official ports.

So I retrieved the last Makefile from openbsd-wip git history and adapt it for latest Hotot version. Source is the zip version from this git revision (latest on 2013-09-09)

The short version (maybe this works only on -current):

    pkg_add http://download.alteroot.org/openbsd/hotot/Hotot-0.9.8.14.tgz
    Hotot-0.9.8.14: ok

Et voila :)

To compile it locally (e.g for openbsd5.3) :

    In your /etc/mk.conf, add /home/user/build/ to your PORTSDIR_PATH variable
    mkdir /home/user/build/net/
    cd /home/user/build/net/
    wget http://download.alteroot.org/openbsd/hotot/hotot-build.tar.gz
    tar -tzf Hotot-build.tar.gz
    cd hotot
    make
    make install

Enjoy :-)

Setup de KVM/Libvirt sur une dedibox

Fri, 06 Sep 2013 00:00:00 +0000

Cet article explique le setup de KVM/Libvirt sur une dedibox de chez online.net, sur un serveur hôte en Debian Wheezy. Évidement, une bonne partie des infos sont valides pour n’importe quel hébergeur.

On commence par installer la base :

    apt-get install libvirt-bin bridge-utils qemu-kvm virtinst

On setup un LVM que l’on appelle “system”, sur lequel on créé des LV pour les VM (je vous laisse vous debrouiller).

Il faut commander une IP Failover et sa mac associée, de type KVM, sur console.online.net et l’assigner à votre serveur. On prendra les paramètres suivants :

    IP hote : 4.3.2.10
    IP VM : 1.2.3.4
    Mac VM : 54:52:00:00:00:00 
    Nom de la VM : vm1

On édite la configuration réseau afin de créer un bridge sur lequel les VM seront attachée, et ajouter une route pour que la VM ai une connexion.

    vim /etc/network/interfaces

    auto eth0
    iface eth0 inet static
      address 4.3.2.10
      netmask 255.255.255.0
      gateway 4.3.2.1
      up brctl addbr dmz0
      up brctl setfd dmz0 0
      up brctl stp dmz0 on
      up ifconfig dmz0 up
      up /sbin/route add 1.2.3.4 dev dmz0

Un petit /etc/init.d/networking restart et ça devrait passer.

Faites un “brctl show”, vous devriez voir le bridge.

Ne pas oublier d’activer l’ip_forward sur l’hôte.

Note : Bien qu’il ne devrait plus être indispensable d’activer le proxy arp si on a fait générer une adresse mac sur l’interface d’online, j’ai tout de même eu des soucis, donc activons le !

    vim /etc/sysctl.conf

      net.ipv4.conf.eth0.proxy_arp=1
      net.ipv4.ip_forward=1

Tout d’abord, il faut créer le LV associé avant de lancer l’installation, puis on utilise virt-install pour la suite :

    lvcreate -v -L 20G -n vm1 system
    virt-install -v --os-type=linux -n vm1 -r 2048 --disk=/dev/system/vm1 --vnc --vnclisten=4.3.2.10 --network bridge=dmz0,mac=54:52:00:00:00:00 --cdrom=debian-7.1.0-amd64-i386-netinst.iso

Connectez vous avec vncview sur l’IP de votre hôte et faire une installation tout ce qu’il y a de plus basique (sans réseau). Rebootez la VM, reconnectez vous en VNC, ensuite place à quelques modifications dans la VM afin d’avoir une console et du réseau.

Pour pouvoir acceder à votre VM avec un virsh console vm1 depuis l’hôte (utile en cas de dépanage…), éditez dans la vm

    vim /etc/default/grub

      GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200"

    vim Editer /etc/inittab
      # Decommenter la ligne suivante 
      T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100

Puis update-grub pour mettre à jour le grub.cfg

Pour le réseau

    vim /etc/network/interfaces

    auto eth0
    iface eth0 inet static 
      address 1.2.3.4 # Ip de votre VM
      netmask 255.255.255.255 # oui, un /32
      post-up /sbin/ip route add 4.3.2.10/32 dev eth0
      post-up /sbin/ip route add default via 4.3.2.10 # Oui, c'est l'IP de l'hote !

Faire un halt de la VM, puis éditez la configuration

    virsh shutdown vm1
    virsh edit vm1

Supprimez les lignes dans le XML à propos de vnc pour desactiver VNC

     <graphics type='vnc' listen type='address' address='4.3.2.10'/> >

On start la VM

     virsh start vm1

La console devrait être disponible

     virsh console vm1

Et la machine devrait pinguer sur le réseau, et avoir accès au net :)

Prochaine étape, activer les drivers virtio pour de meilleurs perfs, dans un autre article

Trucs et astuces :

Il est possible de monter les partitions de la VM sur l’hôte (VM éteinte) afin de pouvoir éditer directement les fichiers sans passer par la console/vnc qui peut parfois être récalcitrant. Pour cela, utiliser kpartx pour activer ou désactiver les partitions du LV !

    # Lister les partitions du LV
    kpartx -l /dev/system/vm1

    # Activer les partitions 
    kpartx -av /dev/system/vm1

      add map system-vm11 (254:9): 0 19529728 linear /dev/system/vm1 2048
      add map system-vm12 (254:10): 0 399894530 linear /dev/system/vm1 19533822
      add map system-vm15 : 0 1951744 linear 254:10 2
      add map system-vm16 : 0 397940736 linear 254:10 1953794


    # On peut ensuite mounter une partition !
    mount /dev/mapper/system-vm1 /mnt

    # Ne pas oublier de umount et desactiver les partitions après opération
    umount /mnt
    kpartx -dv /dev/system/vm1

Use Irc Server-Time capabilities with znc and hexchat/weechat

Tue, 03 Sep 2013 00:00:00 +0000

Server time is a “new” capabilities for IRC protocol (based on work by ircv3 working group, see http://ircv3.atheme.org/ and http://ircv3.atheme.org/extensions/server-time-3.2 ).

If you already use ZNC, you probably use the backlog feature, which allow znc to buffer all messages on irc while you are away, and sent back to you client when you reconnect (yes, I know, you can use screen/tmux with irssi ;-)).

The IRC protocol does not communicate time, however, and a client would typically attribute these buffered messages to the time the client received them. This will always be the time at which the client connected, rather than the actual time the message was sent.

Server-time solve this problem, and is available in hexchat (>2.9.5), weechat (>0.4.0), limechat (> 2.29) and znc (>=1.0).

You just need to activate it on client side.

In hexchat via Settings -> preferences -> advanced -> Use server time if supported

In weechat, /set irc.server.$yournetwork.capabilities “znc.in/server-time” (or “znc.in/server-time-iso” with znc >1.0)

On connexion, you will see something like :

    irc: client capability, server supports: userhost-in-names multi-prefix znc.in/server-time 
    irc: client capability, requesting: znc.in/server-time
    irc: client capability, enabled: znc.in/server-time

Now your backlogs have correct timestamps !

Alteroot:~#

New blog engine!

Xfce infrastructure in 2020

Road to Xfce 4.14, part 2

Core

Xfce Apps

Xfce Panel plugins

Tests

Infrastructure

Git / Bugzilla

Website

Misc

Road to Xfce 4.14

Compte rendu du Meetup ArchLinux France du 10 Nov. 2015

Enhance Apache Cassandra Logging

Working OCSP Stapling setup with free Wosign Certificate on Nginx

Awesome Weekly Newsletters !

DevOps

DevOps Weekly newsletter

DevOps newsletter by ServerDensity

SysAdmin Casts newsletter

Servers for hackers newsletter

Docker Weekly newsletter

Kubernetes Weekly newsletter

UsingChef Weekly newsletter

Briefs on Ansible

Web Operations Weekly

Cloud Development Weekly

This Week on Domains

Development

Python Weekly

PyCoders Weekly newsletter

Ruby Weekly

Go Newsletter

DBs

DB Weekly

Weekly MySQL Newsletter

Postgres Weekly newsletter

NoSQL Weekly

(big)Data Science

Big Data Weekly newsletter

DataScience Weekly newsletter

Web Development

HTML5 Weekly

JavaScript Weekly

Mobile Web Weekly

Node Weekly

Others

RasPi Weekly

Bitcoin Weekly

How to change Cassandra compaction strategy on a production cluster

HTTP alternative services and opportunistic encryption

News from Xfce 4.12, part 3

Core Apps :

Apps :

Panel Plugins :

News from xfce 4.12, part 2 !

Slides du Premier Meetup Archlinux France

Nginx - create a cache based on upstream response time

Remote code execution through bash CVE-2014-6271

News from Xfce !

Chiffrement entre serveurs avec starttls et postfix

Fail2ban on FreeBSD

Archlinux.fr passé, présent, futur

Replace a dead node in Cassandra

Mise en place de DKIM avec Postfix

Xfce - Xfwm4 zoom mode in 4.12

Use KDEConnect with android and Xfce

Playing with the new ownCloud share API

Mise en place d'un serveur mail - Postfix, Dspam et Dovecot

Share on Wallabag from Netvibes

Setup de Mozilla Sync Server

Hotot twitter client on OpenBSD

Setup de KVM/Libvirt sur une dedibox

Trucs et astuces :

Use Irc Server-Time capabilities with znc and hexchat/weechat