Author Topic: I think I'm looking to get this site off my hands  (Read 11710 times)

0 Members and 1 Guest are viewing this topic.

October 27, 2008, 05:31:18 pm
Read 11710 times

JohnC

  • Special Members
  • Hero Member

  • Offline
  • *

  • 1964
It has been a little over a year since I moved this site on to paid hosting. And while updates haven't been everyday, I've pretty much been doing work for the site everyday, whether it is research in some way or another. The idea originally was to have a list which was as dynamic as possible for researchers and other enthusiasts. Sadly, time consumption is a big problem and keeping track of dead sites to remove them took a backseat to adding new sites to the list.

The site has some people that use the list for non-commercial purposes. The site has no contractual obligations. It has no debts, but also no revenue stream because I have avoided putting advertisements on the site. I was in talks a while back to license the domain list to a company, but talks stalled because I wasn't aware the future of the site. So I felt it was futile to continue discussions. I can't discuss anymore details regarding that, like the company involved and figures related to it, since I have a non-disclosure agreement signed.

We don't get a large amount of traffic, mainly because we're not a site like safeweb/siteadvisor, we're aimed at a more technical audience. And we were removed from googles search engines results pages for some reason (now we are back) http://www.malwaredomainlist.com/forums/index.php?topic=1625.0







Some companies often would like information sharing but the sharing seems to be a one way stream. Or if you are entitled to data in return it is to remain private which of course is useless to a site like this. Not to mention problems I have had along the way, like a lack of funding to hire developers because of a lack in programming ability/time. Having problems with my personal PC, like motherboard failing, hard drive failing and losing a lot of research. A problem with Windows Update which still isn't fixed. Which shows no updates to grab but still has one that isn't updated. Which is actually critical and can cause you to be infected.

The site does have some regular users which help out and contribute domains (thanks guys ;) ).

And there is an interest in the site being developed further with additional features added such as:

Email updates
RSS feeds/updates
CSV format
Adding ASN to the list


October 27, 2008, 05:40:19 pm
Reply #1

MysteryFCM

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 1693
  • Personal Text
    Phishing Phanatic
    • I.T. Mate
I'll be happy to help where I can :)

Losing this site, isn't an option ;)
Regards

Steven Burn
I.T. Mate / hpHosts
it-mate.co.uk / hosts-file.net

October 27, 2008, 07:18:58 pm
Reply #2

sowhat-x

  • Guest
Quote
And there is an interest in the site being developed further with additional features added such as:

Email updates
RSS feeds/updates
CSV format
Adding ASN to the list

During these 15 months that MDL is up and running...
After having faced numerous kinds of difficulties as already mentioned,
MDL's dedicated members...most of them being spare-time volunteer researchers as well,
have worked way more than hard I would dare saying...
everyone via his own method and abilities,and altogether in a community spirit.
All in all...I think that no one out has a doubt that we did a pretty good work,
against craphead malware authors out there.

So,to my poor point of view,when talking about people/companies...
that earn money by using the data supplied here in a daily basis,and more over,
without even showing that they're willing to contribute something back,then...
how in the world is it possible to also have...further requests about MDL's data?
Free work for...others?No money,no honey -> simple as that.Just my two cents...

October 27, 2008, 07:31:57 pm
Reply #3

SysAdMini

  • Administrator
  • Hero Member

  • Offline
  • *****

  • 3335
I like this place very much. Its members are a very good team.
It's like a small family. I think it would destroy this wonderful community
if this site would get in commercial hands.
I can only speak for myself - I'm not willing to contribute for a company.

Can we do anything to prevent this ?

Ruining the bad guy's day

October 27, 2008, 10:38:40 pm
Reply #4

sowhat-x

  • Guest
Quote
I think it would destroy this wonderful community if this site would get in commercial hands.

This depends on lots of factors...
one of the most important ones being,the way a company makes business around...
eg.if they're doing it merely for the...cash,or if they actually believe in the fight against crime/malware...
If companies have already shown interest,this means they certainly don't intend of 'destroying' it to say so...

Yet though,I wouldn't be surprized at all,if some people/companies out there,
supposedly for "pseudo-ethical" reasons,
(towards providing malware links and further info in a semi-public way),
don't really like us that much to say so...
kinda of,whoever is not with us,belongs to the...enemy's side,heh...

October 27, 2008, 11:33:56 pm
Reply #5

JohnC

  • Special Members
  • Hero Member

  • Offline
  • *

  • 1964
CSV format: http://www.malwaredomainlist.com/mdlcsv.php

Not sure if it is 100% correct. Some of the columns have commas, which means the fields need to be enclosed in double quotes, but some of the fields contained double quotes. On the wikipedia page for the CSV file format. It said the part with quotes should have an extra set of double quotes outside of it. And when I checked how PHPMyAdmin did it, if I chose to export CSV it gave no extra quotes, and if I used CSV Excel it gave like 4 sets of quotes in total.

So it just has quotes around the field and no extra quotes inside. It all depends on how you parse the CSV file anyway.

October 28, 2008, 12:27:42 am
Reply #6

B_H

  • Special Members
  • Full Member

  • Offline
  • *

  • 49
well, tonight we do small research on cvs file for more detail on GEO location hosts .

 philipp help me for make this output , big tanx ! ;)

B_H

November 03, 2008, 06:23:00 pm
Reply #7

pnuemo

  • Jr. Member

  • Offline
  • **

  • 11
Myself and lithium of MDB are willing to help out in any way we can.  Don't hesitate to PM and ask for anything.  If we can accommodate, we gladly will.

November 19, 2008, 07:04:13 am
Reply #8

hhhobbit

  • Special Access
  • Full Member

  • Offline
  • *

  • 54
Well, I am not commercial despite the http://www.SecureMecca.com domain name.  My stuff all is GNU licensed. I also own SecureMecca.net, SecureMecca.biz, SecureMecca.org (what I probably should have used), SecureMecca.us, and SecureMecca.info.  As soon as some more translation is done (very difficult as opposed to just thinking in French) into French I hope to have either SecureMecca.fr (most likely) or SecureMecca.com.fr (least likely).  I am paying for all of them out of my own pocket.  I don't intend to ever host on most of those names - I just don't want somebody else pretending to be me.

Instead of money (I am really poor) I can give you the following pieces of code that may help.  But before I do that, would you prefer it if I just stripped out all of the dead / parked hosts and put it up in the folder (assuming you had a file today (18 Nov) and I did it to it, the file name would be:

http://www.securemecca.com/MalwareDomainList/hosts_2008_11_18.txt

If you don't want that, the code I am giving you is all designed to work on Unix / Linux since that is what I work on.  Here is where the files are at (you will have to do a make):

http://www.securemecca.com/MalwareDomainList/QuickRemove.7z
http://www.securemecca.com/MalwareDomainList/QuickRemove.zip

ckaaa.c:
======
Checks that ALL of my merged database is in strict ascending order.  Used with duplin to make sure I have something for a program that follows.

ckdupe.c:
=======
Checks for duplicates in a host file, or optionally spits out the names.  I do not store my database in a finished hosts file.  Instead they look like this (and I merge and ssort things into the order I want when I build the hosts files with automated programs).

10000hits.net
10000hits.net (WWW)
10006.hittail.com
100webads.com
100webads.com (WWW)
100webads.com (WWW2)

etcetera.  I wished I hadn't done the www2.* that way but I am stuck with it now.  I have several files going into making my hosts files up:  add.Casino, add.Dead, add.Header, add.PacProxy, add.Porn, add.Proxy, add.WinRisk, and main.  add.Dead hosts are old dead proxies.  I have to block proxies because they effectively turn all filtration off.  The Unix file consists of add.Header (always at the start) main, add.Casino, add.Porn, and add.Proxy.  The Windows file only adds add.WinRisk.  I hesitate giving you the rest of the programs (disorder.c, 2lnx.c, 2win.c, ctrlm, addm and two scripts named newadds.sh and pushhosts) that massage them since you can probably come up with something in PERL that will do those things nicely.  It is just that sometimes I work with some pretty big files (Airelle's files).  I can't use an interpreted language.  It is just too slow.  My scripts that do DNS checks just bite off 100 hosts at a time and rest between queries (using head and tail and sleep).  I should probably write a program for OpenDNS dead hosts, but for now I just use MicroEMACS macros to move hosts that are mapped to those addresses out of the Alive file into the Dead file.  Here are the files:

ferret.c:
======
This is used in conjunction with some of the other files.  It basically splits one set of hosts you have with another set you are looking at to see which of the others you already have or not. There will be an example of using this at the end.

hexcmp.c:
========
A bonus.  I use it with scripts to make sure what ever I uploaded went up safely.  I also used it to construct my own hand-crafted GRUB boot files and patches for the starts of disk (with dd), etcetera.

isparked.c:
========
This contains all of the known parked addresses I have.  The problem is, I keep adding several every month.  Be careful with GoDaddy and some others - they mix both live and parked hosts on the same web server (same IP address).  It drives you nuts, but what can you do?  You just have to check them manually (I use wget for speed - no time for browsers most of the time).

mytmp.c/mytmp.h:
==============
Just comes up with a temporary file name in the current directory.  If you want to be a peach, modify it so the temporary file is put in the folder where the modified file is (you cannot link across file systems).  It is used by serveral of these programs and some more.  I haven't had time to modify it ever since I started working at a good-will store 40 hours per week and work on my web sites / filter 50+ hours per week.

ssort.c:
======
This is a strict ascending sort using the heapsort algorithm.  The reason it exists is because I use its output for a known list as input to ferret which can then use a binary look-up for speed.

HOW TO USE THEM:
================
Okay, I gave you the list of dead hosts in the file say, 2008_10_23_rmlist.txt.  Now that file is sorted so that they are in that order above for my files.  I just used MicroEMACS (thanks Dan Lawrence) to massage them to a format you may prefer.  So here is what I would do, using your hosts file (I am assuming you compiled and put the files in your own home bin directory or some place else in your path):

$  ssort < 2008_10_23_rmlist.txt > AAA
$  duplin AAA
# no output means you are okay.  If you aren't then
$  duplin -s AAA
# you can use uniq but at the time I wrote this program I was using Windows and programming for the Hobbit chip ON DOS
$  ckaaa
$  ckdupe -p hosts.txt > tmp
$  ferret

Okay, now you have two new files, "found" and "okay".  The ones in the "found" file were dead (or maybe Parked - I can give you those in a list as well since I retain a Host <---> IP database (with IPS in the format ###.###.###.### - left pad filled with zeros if necessary) and the ones in the "okay" file are ready to go back into a new hosts file as long as you prepend a "127.0.0.1" in front of them.  That was what my 2lnx and 2win files do.  Despite how simple it looks, ferret is a very powerful program!.  You also need to be careful for the IP addresses 0.0.0.0 (000.000.000.000), 127.0.0.1 (127.000.000.001), and 255.255.255.255.  Beats me what they mean for the last one coming back from a DNS server.  It is certainly unroutable by every protocol I know (RIP, RIPv2, EIGRP, OSPF)!  Maybe that is the point.  If some idiot is setting up some routers, the 127.0.0.1 IP address is NOT turned off by default as are the three NRIP address spaces.  You have to turn all of them off in the configuration on some routers (actually most for the NRIPs).

But you can also turn around and use all you have in the AAA file, and the new ones you are considering adding in the tmp file and ferret will spit out the host names you don't have in the "okay" file.  Like I said, ferret is a very powerful program.  It spit the stuff out so darn quick when I first wrote it I began to wonder if it was functioning okay.  It is functioning okay!  It is just blisteringly fast!

I can't give you any money since I only make a minimal amount of money at the charity store and after seeing an idiot (a customer) yesterday pulling apart a VCR / DVD combo with the power plugged in I figure I better get out of there.  In fact, I left for the day when the managers said I was wrong to yell at him for destroying merchandise.  I also have a school loan to pay off so money for me is scarce.  I will help how ever I can though.  The problem is I am swamped with what I have to do.

December 18, 2008, 09:43:22 pm
Reply #9

Kayrac

  • Guest
Hey john, i was wondering what ever came of your decision?, if anything?