H4ck3rm1k3's Blog

exploration of free/libre open source software and things

Forking python tests : forkr — July 23, 2017

Forking python tests : forkr

Often we work on large static datasets and code that takes a long time to load. Then we work on adding on little bits of code on top of that, but the time to load the data needed to run a test case can be daunting.

In such cases we can load all the data we want and then fork the process, the child process will inherit all the variables from the parent and you can then load new snippets of code on top of that baseline to do you development.

For this I created the forkr lib that you include with

import forkr

forkr.main(globaldata)

The global data will be made available to the child process.

I find running the nose test modules in python very useful. The only problem is getting the global data into there, so I created a simple plugin that you can use forkr.DataPlugin, this overrides the prepareTestCase to install the __global_test_data__ variable into each nose test module that you load, this way tou can

Code is here: https://github.com/h4ck3rm1k3/py-loadr-forkr-debugr/blob/master/forkr.py tested on python2.7, compiles on 3, not tested yet

Python supports reloading code, but I have not been able to get that working to my satisfaction.  Best is to load all your variable code in the child process.

Advertisements
Installing git on sdf — December 17, 2016

Installing git on sdf

In some cases you want to install git where you are only a user,  the .bashrc is not being interpreted, and sshd does not allow you to send the path variables, in short a locked down system. What you do in this case, you unpack the binary packages into some path, and prepend that path via a git ssh command script to your ssh command. Here we do this for sdf, the superdimensional fortress that gives you lifetime shell and web hosting for $36 one time fee (http://sdf.lonestar.org/index.cgi?access). It is running netbsd 6.0 on an amd64 machine , so we fetch the binaries right off the server.

How to install git on sdf as a arpa member :

  1.       mkdir ~.local
          cd ~.local/
          wget http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/6.0/All/git-base-2.10.0.tgz
          wget http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/6.0/All/git-2.10.0.tgz
          tar -xzf git-2.10.0.tgz
          tar -xzf git-base-2.10.0.tgz
          rm *.tgz
        
  2. On your client machine :
      1. Setup your ~/.ssh/config
      2. Host sdf
        	  User h4ck3rm1k3 
        	  HostName sdf.org
        	
      3. Setup ssh login script to prepend path
        	    #!/bin/bash
        	    
        	    shift # remove the hostname that is arg1
        
        	    # now we set the path and call the command without the hostname
        	    # host name is sdf that is configured in the .ssh/config
        	    exec /usr/bin/ssh sdf "export PATH=/sdf/arpa/gm/h/h4ck3rm1k3/.local/libexec/git-core/:/sdf/arpa/gm/h/h4ck3rm1k3/.local/bin:/usr/pkg/bin:/usr/local/bin:/usr/bin:/bin;$@"
        	    
        	  
      4. Setup your git config to use custom login script
        	    [core]
        	    repositoryformatversion = 0
        	    filemode = true
        	    bare = false
        	    logallrefupdates = true
        	    sshCommand=/home/mdupont/.ssh/sdfgit.sh
        	    [remote "origin"]
        	    uploadpack = /sdf/arpa/gm/h/h4ck3rm1k3/.local/libexec/git-core/git-upload-pack
        	    url = ssh://sdf:/sdf/arpa/gm/h/h4ck3rm1k3/html
        	  

Sources

  1. https://git-scm.com/docs/git-config
  2. https://linux.die.net/man/5/ssh_config
  3. http://sdf.org/
Please help with school district map for Hamilton, NJ — September 3, 2016

Please help with school district map for Hamilton, NJ

We (http://codefortrenton.org/) are working on a project to create a school district map for Mercer Country NJ.

Our map so far is here: http://codefortrenton.github.io/school-districts/mercer/#9/40.1074/-74.5313

We are still missing the data for Hamilton because the data we got was in a low res pdf file that was a screen print of the original gis data. It is also up side down etc.

Here is the original pdf : https://github.com/codefortrenton/school-districts/blob/master/Hamilton/Hamilton.pdf

We split out the pages into png and are working on rectifing them with mapwarpers

The pngs are here https://github.com/codefortrenton/school-districts/tree/master/Hamilton

Example:

Greenwood

Example rectified map : http://www.mapwarper.net/maps/15521#Export_tab

and resulting OSM relation https://www.openstreetmap.org/relation/6535002

We created a Google doc to track progress so far,

The sheet with with the doc is here, leave a comment on it and we will add you to the writers.

https://docs.google.com/spreadsheets/d/1MM-wjl1Nw_efo0ty9ZkZtT7247H-Q7R7JChReTZLcPA/edit#gid=0

please help !

If you want to chat, you can join our slack channel https://code4trenton.slack.com/messages/opendata/, and if you don’t like slack I am alone in the irc.freenode.net #codefortrenton channel

1732 SIXTH ST, Ewing NJ is smaller than listed — August 27, 2016

1732 SIXTH ST, Ewing NJ is smaller than listed

The house located at 1732 SIXTH ST Ewing NJ is listed falsely on the county website and hud home store.

We were interested in the property and upon inspection measured the length of the house from front to back, it was 41 feet long. The county website lists it as 48 feet long.

hudhomestore The house is listed there at 939 square feet.

Screenshot from 2016-08-27 07-15-16

On the mercer county site it is listed as 1128 square feet. http://pip.mercercounty.org/propertycard/1102_52_203.pdf

Screenshot from 2016-08-27 07-07-32

The actual size of the house is different, it is not 48 feet long, but 41 feet on the exterior, we calculated the total space from outside to be 888 sqft max. This does not count for walls,

The front part A is 17 * 20 feet, not 24 * 20 in size. That is a difference of 240 sqft so the tax information would be incorrect.

 

 

Lawrence School Districts — August 14, 2016

Lawrence School Districts

Here is another take on the Lawrence Township School Districts,

I took the parcel outlines from mercer county and associated them to schools via a ruby script, as an output we have a set of shape files.

lawrence3

Then I converted them to points using qgis “extract nodes”

 

lawrence2

Then I used the voronoi polygons on them to calculate regions,

lawrence4

and clipped it with buffered region

Finally I dissolved all the borders between the school regions to give the simple borders:

lawrenceFinally I used the qgis web export to create a website :

(export to web map)

http://codefortrenton.github.io/school-districts/mercer/lawrence/

You can find my scripts to convert and look up the data here:

https://github.com/h4ck3rm1k3/shawnee-county-osm-landuse

and the data for the school districts here:

https://github.com/codefortrenton/school-districts

Need a way to find speakers for open source conferences — July 27, 2016

Need a way to find speakers for open source conferences

Hi,

I am working on the next edition of the Software Freedom Kosova Conference on the 22nd of Oct, and we are looking for speakers (http://sfk16.flossk.org/call-for-speakers).

I am thinking that we need a way for potential speakers to register and make themselves available to conferences. They would register with a generic system, define the topics they can speak about (maybe pull from github and gitlab commits etc to show abilities) and then define locations and times available.

Conference organizers would register and be vetted (to prevent spam) and after they check out speakers could be contacted via the system and invited to speak.

Commission on English Language Program Accreditation whitewashes history — April 13, 2016

Commission on English Language Program Accreditation whitewashes history

The webpage of Commission on English Language Program Accreditation has tried to whitewash the internet.

http://www.cea-accredit.org/about-cea/news/jan-2014

Previously they had the college of northern NJ listed:

Granted 5-year initial accreditation
  • Woods English Language Institute, University of Northern New Jersey, NJ, USA

But this was removed.

I have made backups of the page previously.

archive.org snapshot of page

http://archive.is/P9tz4

http://www.freezepage.com/1459984414CHSHKXXITX

http://www.shrinktheweb.com/reqstatus?status=1&hash=2dcf14d3780f5b09a086421baca168d2

 

Proposal for Civil Solution System — April 5, 2016

Proposal for Civil Solution System

Cooperation

I have been thinking a long time about what we need for civil solution planning and implementation. I propose we create an open source platform and build a platform that can be used by many people in many places. The basic idea is that citizens can raise issues, collaborate on them, select the ones that are important and the define what is needed to implement them, and finally track pledges and deliver pledges to implement the problems. Government agencies fit into the system because they are already pledging to provide services and we will ask them to deliver on these promises. We want to support civil funding of solutions via money or even services, work, food, transport or shelter etc to solve problems.

What are the activities supported by this system?

  1. Registration of users via self service or via phone messages or other means. Allow users to login via fb or twitter.
  2. Storing and Handling of uploads of geotagged pictures of problems. Allow for users to position the photos. Allow multiple photos per incident.
  3. Allow for users to send sms and mms to a phone number
  4. Collection of problem description and severity
  5. Sharing of problems with other user via social media
  6. Collection of votes and comments and feedback from other users to help sort and collaborate on problems.
  7. Reporting of problems via Fax, PDF file or statistics to responsible agency
  8. Tracking of response and feedback from agency
  9. Tracking of progress of resolution, if the agency does not respond setting timeouts and limits.
  10. Display of map with easy reporting of problems.
  11. Creating of mobile apps that allow the easy reporting of problems.
  12. Marketing of the application via traditional media
  13. Allow people to pledge money or resources for the resolution of the resources
  14. Allow for volunteers to donate time to help process requests.
Why is this project important?
  1. Serving the people better
  2. Create a platform for managing volunteer resources
  3. Managing the funding of citizen problems.
  4. System accessible via simple sms or phone
  5. Increasing of transparency of government
  6. Easy Tracking of citizen reports
  7. Tracking of agency response times
  8. Prioritization of problems via group feedback
  9. Need a central server for Hosting citizen feedback.

Implementation, A synthesis of the following software :
https://www.whatdotheyknow.com/
https://www.ushahidi.com/
http://www.sugarcrm.com/
http://www.selfstarter.us/

We would implement an http://www.open311.org/ like system.

and various projects from https://www.codeforamerica.org/

See my older post on using social media for conference organizing that contains some of the ideas covered here in a different context.

UPS store overcharges you to print from internet — March 27, 2016

UPS store overcharges you to print from internet

Yesterday at around 4pm, we went to the UPS Store at 957 NJ-33 #12, Trenton, NJ 08690 to print a few pages and notarize them. The refused to type in a url to print for us and gave us no other option except to email them or use the computer at a minimum charge of $2.50. After we got home I found out I can print 6 pages for $0.12 via HP eprint and have it printed at the UPS store,  http://compass.ups.com/print-from-your-mobile-device-at-the-ups-store/ They did not offer me this or tell me about it. I feel ripped off.

 

bambam mods for kids — March 20, 2016

bambam mods for kids

Bambam is a great game for kids  but it only contains a few pictures

https://github.com/porridge/bambam

my son likes a particular tv show, and I found a way to download images from those characters into bambam.

here are the steps,

  1. get the images with the wikiteam tool https://github.com/WikiTeam/wikiteam

    ./dumpgenerator.py –images http://paw-patrol.wikia.com/wiki/PAW_Patrol_Wiki

  2. convert them into gif

    for file in *; do convert $file “$file.gif”; done

  3. resize them (or buy lots of ram), because bambam loads them all

    mogrify -resize 256×256 *.gif

  4. move them to the data directory of bambam
  5. run!