openssl s_client -connect www.cyberciti.biz:443
As the included diagram portrays, in a common workflow, the worker instance will consume messages sent to a specified Amazon SQS from another service (e.g.: a web server or another worker). These messages will be received by the worker via POST requests. This eliminates the necessity of configuring a worker as an always-on service, as well as having to add code for reading and consuming messages from an AWS SQS queue. In other words, the worker is implemented as a standard RESTful API/Service that will react to a message sent to it at an specific endpoint via a POST request. This is an awesome approach by Amazon to microservices and reactive design.
The conversion of the SQS message to a POST request is executed by what AWS calls the "SQS Daemon" or "Sqsd". This is a simple daemon they pre-install in the worker tier instances that is constantly monitoring an specific AWS SQS queue (provided by configuration) for new messages. When new messages arrive, it constructs a POST request and sends it to a specific endpoint (also provided via configuration). If the endpoint consumes it without errors and returns a 2** HTTP Code in its response, the "Sqsd" deletes the message from the queue to signal that its consumption was successful.
However, even though this approach is extremely powerful, Amazon does not provide the code of this daemon as open source. Therefore, we have reproduced its behavior by creating our own version of the "Sqsd" free for everyone to use. Moreover, we have provided lots of customization and configuration properties so that it can be molded to your specific use cases.
Don't give s3 full access policy to your app user
Prefer to allow access only for specific bucket
I wonder what's the best choice : create managed policy or simply use inline policy. I got a 1 to 1 relationship between my app-users and bucket so... inline policy looks good here
spoiler alert: auto scaling is far to be magical
as usual with aws, everything is an object and for autoscaling u got several objects and several links between them
First you define a launch configuration : what type of machine you want to laucnh
Then an autoscaling group : it will use the launch configuration to create new EC2
In this autoscaling group, you have to define auto scaling policies ie what to do (remove or add x instances) and link them to a cloudwatch alert (cpu is high or network or whatever cloudwatch monitors)
the autoscaling group can also be linked to an ELB so when EC2 are added/removed, they also are registred/deregistered from ELB
A lot of work to do..
Example : I change the autoscaling trigger in web UI interface. After a few second, my autoscaling trigger is changed (checked).
But if I save beanstalk environment configuration into a file, the change I made is not there :-(
Maybe working wih UI is not the best option, but it's very convenient to test stuff without terminate/recreate env from scratch (things I have to do btw when I commit changes in files, I have to check if the behaviour is the same)
A lot of time lost, but it's the price to have reproductible environment and 'infrastructure as a code'
Each time you start a stopped instance we charge a full instance hour, even if you make this transition multiple times within a single hour.
I'm happy with vimdiff (colorscheme Murphy) but why not!
Elastic Beanstalk simplifies this process by managing the Amazon SQS queue and running a daemon process on each instance that reads from the queue for you. When the daemon pulls an item from the queue, it sends an HTTP POST request locally to http://localhost/ with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST. You can configure the daemon to post to a different path, use a MIME type other than application/JSON, connect to an existing queue, or customize connections, timeouts, and retries.
Cycle de vie des messages Amazon SQS
Les messages stockés dans Amazon SQS ont un cycle de vie facile à gérer, mais qui garantit le traitement de tous les messages.
Un système qui doit envoyer un message sélectionne une file d'attente Amazon SQS et utilise SendMessage pour lui envoyer un nouveau message.
Un autre système traitant des messages doit traiter plus de messages : il appelle donc ReceiveMessage, et ce message est renvoyé.
Dès lors qu'un message est renvoyé par ReceiveMessage, il ne sera pas renvoyé par une autre demande ReceiveMessage avant que le délai de visibilité ne soit expiré. Ainsi, plusieurs destinataires peuvent traiter le même message simultanément.
Si le système de traitement des messages termine avec succès le traitement de ce message, il appelle DeleteMessage, ce qui supprime le message de la file d'attente pour que personne d'autre ne le traite. Si ce système ne réussit pas à traiter le message, il sera alors lu par un autre appel ReceiveMessage dès que le délai de visibilité sera expiré.
Si vous avez associé une file d'attente de lettre morte à une file d'attente source, les messages seront déplacés vers la file d'attente de lettre morte lorsque le nombre de tentatives d'envoi que vous avez défini aura été atteint.
Running Bottle with a different server
As said above, the standard server is perfectly suitable for development, personal use or a small group of people only using your application based on Bottle. For larger tasks, the standard server may become a bottleneck, as it is single-threaded, thus it can only serve one request at a time.
But Bottle has already various adapters to multi-threaded servers on board, which perform better on higher load. Bottle supports Cherrypy, Fapws3, Flup and Paste.
If you want to run for example Bottle with the Paste server, use the following code:
from bottle import PasteServer
...
run(server=PasteServer)
This works exactly the same way with FlupServer, CherryPyServer and FapwsServer.
import logging
logging.basicConfig(filename='log.txt', format=logging.BASIC_FORMAT)
logging.error('OH NO!')
try:
raise Exception('Foo')
except:
logging.exception("Oops:")
Today I got a docker daemon not responding to commands (docker ps, docker run...)
Even after restarting service
In /var/log/docker I got this:
time="2017-03-01T08:28:58.43946917Z" level=fatal msg="open /var/run/docker/libcontainerd/containerd/81623262351dfc42c5e87aa8df11592a57f2d14a468476620c7c4d6c89de1958/state.json: no such file or directory"
The solution was to stop docker service, remove this /var/run/docker directory and restart docker service
Beanstalk ne permet pas de définir un trigger basé sur le nombre d'éléments dans la queue SQS pour l'auto scaling des environements de type worker
Pourtant c'est possible mais il y a un peu de boulot :
- créer des scaling policies qui vont ajouter ou enlever des instances
- créer des alarmes cloudwatch qui vont se déclencher par rapport au nb d'éléments dans la queue et appeler les scaling policies..
http://docs.aws.amazon.com/autoscaling/latest/userguide/as-using-sqs-queue.html
https://forums.aws.amazon.com/thread.jspa?messageID=722589
On peut surement intégrer ça à beanstalk avec les .ebextensions (http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html)
En plus d'être un peu complexe et pas vraiment intégrer à beanstalk, ça n'a pas l'air très réactif, cet article en parle et donne une solution :
Rapid Auto Scaling with Amazon SQS : https://aws.amazon.com/blogs/aws/auto-scaling-with-sqs/
Or "how to search text in git diff history"
git log -p -S "string"
Depuis hier le 22 Février 2017
seems cool and powerful
Warning : this loop does a lot of call to aws api, use it with caution
To know that I needed to list all policies and associated statements (for the default policy version)
#!/bin/bash
IFS=$'\n'
for line in $(aws iam list-policies|jq '.Policies|.[]|[ .PolicyName, .Arn, .DefaultVersionId ]| @csv' -r|sed 's/","/ /g'|sed 's/"//g'); do
name=$(echo $line|cut -d' ' -f1);
arn=$(echo $line|cut -d' ' -f2);
version=$(echo $line|cut -d' ' -f3);
echo "$name"
aws iam get-policy-version --policy-arn $arn --version-id $version
done
Put this in a script, redirect output to a file and go get grep!
Since December 22th 2016 you are able to configure application version lifecycle in Beanstalk.
Very convenient, we can trash our custom api cleaning scripts :-)
Default listener (80) is enabled by default, to disable it :
aws:elb:listener:
ListenerEnabled: false
docker stop command send SIGTERM to pid1 inside the container to let a chance to stop gracefully (https://www.ctl.io/developers/blog/post/gracefully-stopping-docker-containers/)
Unfortunatly some software ignore SIGTERM and need to be stopped by their own command (hello opendj)
To do so, in the Dockerfile, in the CMD, I use a script like this :
/thecommand/tostart/thesoftware/inbackground
trap "/thecommand/tostop/thesoftware" SIGTERM
while true; do sleep 1; done
so my script is run with PID1, docker stop sends SIGTERM, my script catch (trap) the signal, and run the command to stop gracefully
cat prepare-commit-msg
NAME=$(git branch | grep '' | sed 's/ //'|cut -d'/' -f2|cut -d'-' -f1,2)
echo "$NAME $(cat "$1")" > "$1"
for dir in $(ls); do cp prepare-commit-msg $dir/.git/hooks/prepare-commit-msg; done
text=" [FIX] ()
🔄 [MOD] ()
✅5 [ADD] ()
🔀 [TEST] ()
[DOC] ()"
echo "$text $(cat "$1")" > "$1"
DELIM=$(echo -en "\001");
sed -i "s${DELIM}SEARCH${DELIM}${VAR}${DELIM}" $config
The easiest way would be to find the head commit of the branch as it was immediately before the rebase started in the reflog...
git reflog
and to reset the current branch to it (with the usual caveats about being absolutely sure before reseting with the --hard option).
Suppose the old commit was HEAD@{5} in the ref log:
git reset --hard HEAD@{5}
Pour avoir une vue globale des crins qui tournent sur différents serveurs
TIL what is this offical repository :
Only debian with some useful package to build dependencies (needed when you install some gem for example)
For example ruby:2.3 Dockerfile use a buildpack-deps image
Several ways to get metadata (like ip address) when you're connected to an EC2:
/opt/aws/bin/ec2-metadata
Je me mets ça de côté pour gagner du temps la prochaine fois
logfile=/tmp/logfile.log
errorfile=/tmp/error.log
function log() {
echo -e "\e[34m\e[1m[$(date +%Y-%m-%d) $(date +%H:%M:%S)] $1\e[0m" | tee -a /tmp/logfile.log
}
function logerror() {
echo -e "\e[91m\e[1m[$(date +%Y-%m-%d) $(date +%H:%M:%S)] $1\e[0m" | tee -a /tmp/logfile.log
}
log "prout"
logerror "pwet"
Create a new file, curl-format.txt, and paste in:
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
Make a request:
curl -w "@curl-format.txt" -o /dev/null -s "http://wordpress.com/"
I don't know if this shit is expected (normal behaviour or not) but this is annoying
bash :
$ toto=tata
$ echo $toto
tata
$ toto='tata'
$ echo $toto
tata
$ toto="tata"
$ echo $toto
tata
avec docker run -e
$ docker run -e toto=tata -it debian:jessie bash
root@5de64152ed05:/# echo $toto
tata
$ docker run -e toto='tata' -it debian:jessie bash
root@99a6b62f595c:/# echo $toto
tata
$ docker run -e toto="tata" -it debian:jessie bash
root@ab7581c30734:/# echo $toto
tata
avec docker run --env-file:
$ cat env.file
toto="tata"
$ docker run -it --env-file env.file debian:jessie bash
root@805480de4a52:/# echo $toto
"tata"
$ cat env.file
toto='tata'
$ docker run -it --env-file env.file debian:jessie bash
root@004b88cc448d:/# echo $toto
'tata'
$ cat env.file
toto=tata
$ docker run -it --env-file env.file debian:jessie bash
root@2ca9908d1cdf:/# echo $toto
tata
avec docker compose:
$ cat docker-compose.yml
version: '2'
services:
foo:
image: debian:jessie
container_name: foo
environment:
- toto1=tata
- toto2="tata"
- toto3='tata'
command: tail -f /var/log/dmesg
docker-compose up -d
docker exec -it foo bash
root@71a9a886cc24:/# echo $toto1
tata
root@71a9a886cc24:/# echo $toto2
"tata"
root@71a9a886cc24:/# echo $toto3
'tata'
TIL
Beanstalk is a little bit... magic. Dunno yet if its good or not :
Elastic Beanstalk creates log volumes on the container instance, one for each container, at /var/log/containers/containername. These volumes are named awseb-logs-containername and should be mounted to the location within the container file structure where logs are written.
For example, the following mount point maps the nginx log location in the container to the Elastic Beanstalk–generated volume for the nginx-proxy container.
{
"sourceVolume": "awseb-logs-nginx-proxy",
"containerPath": "/var/log/nginx"
}
And all this logs can be copied into s3 automagically: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-elasticbeanstalkhostmanager
Via Doo
20h d'autonomie annoncée pour celui là :
Lenovo ThinkPad X270
Le Lenovo ThinkPad X270 est conçu pour durer. Grâce à sa batterie, il pourra en effet tenir pas moins de 20 heures avec une seule charge. Son châssis a aussi été renforcé pour lui permettre de résister aux chocs.
Plusieurs configurations seront proposées. L’écran de 12,5 pouces pourra aussi afficher du 720p ou du 1080p. Derrière, on pourra opter pour des Intel Core iX couplés à un maximum de 16 Go de RAM. Côté stockage, Lenovo proposera des disques à plateau (2 To max) ou du SSD (512 Go max). Il faudra cependant se contenter d’un GPU intégré.
Le prix sera fixé à 949 $ pour le modèle de base.
Sympa les petites nouveautés!
docker system df
docker system prune
CLI restructured <3
Improved CLI backwards compatibility <3
cat <<'EOF' >> /tmp/bar
foo
EOF
cat <> /tmp/bar
$foo
EOF
Dashing n'est plus maintenu, voici son fork, Smashing!
Compose file version 2.1 and up
Healthcheck configuration can now be done in the service definition using
the healthcheck parameter
Containers dependencies can now be set up to wait on positive healthchecks
when declared using depends_on. See the documentation for the updated
syntax.