Automated BrickLink cost minimization program available

skywalkerx86

just sent a message to @atxdad to get this program but wanted to post on here to thank the other's that helped him with modifications. Can't wait to get it running so no further head-banging against the wall trying to get good deals on large quantities of bricks

111ins

is there anyone out there thta would run this for me? I would try, but I download family pictures and loose them in my computer, so I do not think this is the best thing for me to try!
compensation offered :)

badbob001

Is there a way to import the Bricklink Wanted List into this script instead of needing to recreate it in BrickStore first? I like to casually browse for parts on Bricklink from my phone and add them to my Wanted List.

It was initially tricky to get the scripts working with Octave. I has associated *.m files with octave.exe but I double-click them or run them from the commandline, the scripts would complain about files not being defined. From an earlier post, I learned that I need to first run octave and then from within it, 'cd' to the script folder and then run the script. Is there a way to invoke the scripts more directly?

atxdad

I don't know of any way to export Wanted Lists from Bricklink, although I have seen that request a few times. If someone can describe a method for doing so, I could probably write a script to parse one into BrickStore .bsx format; it's just plain text format with some markups.

badbob001

The only way I see to 'export' the bricklink wanted list is 'print' it and parse the resulting html page.

1. Go to Wanted List page and click 'Show All'
2. Click the printer icon in the upper right
3. To get everything on one page, add "&wantedSize=100" to the end of the url and go. Adjust the number to accommodate your list size.
4. Save the page as html file

In the html file, the relevant section for each item looks something like this:
<A HREF="catalogItem.asp?P=30374">Yellow Bar 4L (Lightsaber Blade / Wand) </A></B></FONT></TD></TR><TR BGCOLOR="#EEEEEE"><TD ALIGN="CENTER"><FONT FACE="Tahoma,Arial" SIZE="2">New</FONT></TD><TD ALIGN="CENTER"><FONT FACE="Tahoma,Arial" SIZE="2">3</FONT>

The following regular expressions seems to be able to find each instance:
<A HREF="catalogItem.asp\?P=[^>]+">[^<]+</A></B></FONT></TD></TR><TR BGCOLOR="#EEEEEE"><TD ALIGN="CENTER"><FONT FACE="Tahoma,Arial" SIZE="2">[^<]+</FONT></TD><TD ALIGN="CENTER"><FONT FACE="Tahoma,Arial" SIZE="2">[^<]+</FONT>

From the above text, it's possible to extract:
ItemID: 30374
ItemName: 4L (Lightsaber Blade / Wand)
ColorName: Yellow
Category: Bar
Qty:3
Condition:New

The most tricky part I think is parsing out the color, category, and name from the single description text. Also tricky is that if a specific color is not requested, the color text will not be in the description. I suppose one can do a substring search of the description field against a table of known colors and a table of known categories. Another possibility is to look at the 'alt' text of the image tag, which has the item description without the color, count the length of that string, and truncate that number of characters from the main string.
<A ID='imgLink0' HREF='/catalogItemPic.asp?P=30374' REL='blcatimg'><IMG ALT="Part No: 30374 Name: Bar 4L (Lightsaber Blade / Wand)" TITLE="Part No: 30374 Name: Bar 4L (Lightsaber Blade / Wand)" BORDER='0' WIDTH='80' HEIGHT='60' SRC='http://img.bricklink.com/P/3/30374.gif' NAME='img0' ID='img0' onError="killImage('img0');">

It's likely that not all fields are really necessary for the minimizer scripts to work. I'm guessing that from the description field, we really just need to extract the color.

Also note that not all brickstore xml fields are available, like categoryID, so the resulting file may not work correctly back in brickstore without some extra work, if that is important to anyone.

I hope this helps.

atxdad

Per my testing, it seems that BrickStore needs only 5 fields to describe each item when it loads a .bsx file; it will populate all the other fields automatically. It errors during load if it cannot find any of these fields:

ItemID
ItemTypeID -> Manually define as 'P' for PART
ColorID - a number from the color ID lookup table
Qty
Condition - N, U, or absent for undefined

Fortunately this information is available in the strings above. I agree that color name parsing might be difficult, but there is a list of color names and their ID numbers in the distribution package (color table.txt).

Tevans333

This is definitely the one feauture I am still looking for:)

Bosstone100

I'd love to know if there is a way to read and then create another *.bsx in VBA. I have an idea for a program but don't want to say anything until it's finished. If anyone knows, feel free to PM me. Thanks.

graphite

^ of course there is.. There is a way to do whatever you want with a bsx file. A bsx file is just an XML style text file with a different extension. I'm currently working on a command line program to take in a list of wanted lists and a list of orders (imported into brickstore and saved as bsx) that will automatically reduce my wanted lists by whatever was purchased in the orders. I frequently have 3+ projects I'm ordering parts for and its a pain to go back and reduce each wanted list when each order contains parts for various wanted lists.

graphite

atxdad said:

although I have seen that request a few times.

One of the requests may have been me. I think I posted to the project about that feature but from the project data it appeared no one had worked on brickstore in at least a couple years. I downloaded the source and was going to start looking at it to add a few features like that one, but I haven't had time to look into it much.

And it is hard to get motivated on something like that when I do programming for work so coming home and programming more doesn't always excite me.

badbob001

It would also be nice for the script to be able to login to bricklink and automatically download the wanted list. I think it just has to access two pages: the login page and then the wanted list print view.

atxdad

The automation required to login to BrickLink as if via a browser exceeds either Matlab's ability or my experience. I could manually follow the guidance to download the source html for my wanted list though, but it is a parsing nightmare, and I even had a clue above as to what to look for.

LegobrandonCP

Just got the file from @atxdad.

Should I get MatLab or Octave?

I know nothing about either. The scripts located in the zip is confusing me as well.

LegobrandonCP

Looks like I'm not qualified to download the trial MATLAB software without some lying to do.

emilewski

Octave is working fine for me (and the price is right).

Poochy

No problem with Octave here either (just got into playing with this last week). I do recommend getting the GUI Octave program though, as I could not figure out how to get the command line version to work, plus the GUI version is just so much prettier :).

emilewski

The easiest thing to do with command line (assuming Windows version) is to dump all the zip files into the root of your home (C:\users\) and put your .bsx wanted parts list there as well. Then you can open up Octave and type "RUN_ALL" after editing the appropriate .m files for your source file and preferences. If you pick a different directory I had to tell Octave where every time. I am sure the GUI Octave is worth the effort of installing too.

Poochy

Thanks, @emilewski. When I tried double-clicking on octave.exe before, all I got was a blank command line window. I did not realize that you had to wait for a while before seeing the prompt (just tested again now). I was just too impatient before, and though that something was wrong with my setup. With the GUI, it says something like "connecting to Octave", which made me feel better about waiting.

emilewski

Yes, it does take a few moments depending on the speed of your system. In my post above, the formatting got stripped...it was C:\Users\Username not just C:\Users.

Branko

I managed to compile a matlab GUI to an .EXE file quickly as a test.
This can be run independently from Octave or Matlab but it requires a 400Mb library file to be installed.
atxdad, if you want I can assist in creating this package for those interested. (the 400Mb file is provided on the matworks website, your actual project is only 0.5Mb)

mdelleman

I've tried to run Taj Mahal 10189 through the program and have had no luck. I am getting next to no results on it, just a few parts from a combo of 2 or 3 vendors. When I remove parts giving off warnings as no one has quantity need for that part I get no combos at all.

Any ideas?

graphite

^ if you're trying to find 2 or 3 vendors that have the entire parts list from a 5900 part set then that is your problem. Something that large probably couldn't be done in less than 10 vendors but if you try to run the program with 10 vendors it would probably take a couple days to run if not more.

LegobrandonCP

Anyone have a link to where I can download Octave for windows? I Googled it, but cant find anything.

Poochy

^ This is the variant I'm using: http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.4.3 for Windows MinGW Installer/

mdelleman

graphite said:

^ if you're trying to find 2 or 3 vendors that have the entire parts list from a 5900 part set then that is your problem. Something that large probably couldn't be done in less than 10 vendors but if you try to run the program with 10 vendors it would probably take a couple days to run if not more.

How do I change the program to run through more than 3 vendors? Forgive me if this is somewhere in the readme.

Poochy

^ find_minimum_cost.m -
* define the minimum and maximum number of vendors to use when searching for solutions

Please consider reading the whole README.txt though, as it does contain a lot of useful info, in particular how to efficiently search for a large number of lots (i.e. by combining multiple searches).

dougts

^ it's either there or in this thread somewhere. But as stated, anything over about 5 or 6 vendors will result in run times that are mostly unfeasible.

I just BL'd the Taj Mahal and it took me 22 vendors plus LEGO replacement parts.

this process doesn't really handle large sets so well. a good strategy is to first look at your wanted list in BL, find all the parts that are only carried by a few vendors in the quantities needed, can be really expensive, or are generally rare, even in partial quanties. look at each store that has the rare part, and find out what other items in the set they also have at reasonable prices. Then order those parts, and add on other needed parts to the same order, as long as the prices are close to what you could get them for overall. keep doing this until all your rare/expensive parts have been ordered, along with the other items from teh same vendor. this will whittle your list down quite a bit, then you can run the remainder through the scripts to find the best combination for the common/cheap parts

Branko

I noticed someone once mentioned looking into porting this tool. Did anyone every take on that task?

Titus

Hey fellow users of this awesome script. I'm having troubles finding stores that appear in my results. As in, I can't find the store on Bricklink. This has happened to me since I started using the software but it's finally made me crazy enough to ask :)

How do you locate the correct store? I usually do a search for the store name in Bricklink's "Store Name" search but that often doesn't quite work, so then I use a portion of the name I get from Octave and can usually find the right store. However, sometimes it can't find the store at all. For example, can anyone tell me how to find the store "Bricks-r-Kids"? Thanks

DadsAFOL

I had this same issue and Chris advised to delete the vendor_locations.mat file. Apparently this is loaded the first time you run the program and never refreshed. So if you've been using it for months, the link between store number and store name can get corrupted by stores that close or change their name. Deleting the file causes it to just reload a fresh copy.

atxdad

The database of vendor locations does get appended with new store info every time you parse a Price Guide, but it does assume that data already in the database are good. Should a store close and its ID number be reused for another, that won't be detected by the script. DadsAFOL is correct: Deleting the vendor_locations.mat file will clear old data, allowing current content to be saved instead.

LegobrandonCP

I'm still trying to figure out how to download Octave with the link posted by @Poochy. :-/

graphite

http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.4.3 for Windows MinGW Installer/Octave3.4.3_gcc4.5.2-install.exe/download

For windows. If it doesnt start, there is a "direct link" link in the top part of the page in small print.

LegobrandonCP

Thanks! Seems like the first time I downloaded the file, it didn't finish completely.

I have it installed now and when I launch the program, a black window shows up with a bunch of white letters, kind of like Windows CMD. I tried tying in RUN_ALL as stated in the readme.txt file, but it didn't work.

Could someone please PM me giving me the extra details and steps to get this up and running?

Thanks!

Poochy

^ Did you see the octave header and "prompt"?

GNU Octave, version 3.6.1
...
...
octave:1>

LegobrandonCP

^ Mine is Octave 3.4.3, other than that, everything is there.

Poochy

What happened when you type RUN_ALL? Do you get any error messages? You may have to cd into the correct directory. 'pwd' to see where you are at, and then 'cd' to get into directory where you had unzip'ed and placed the files that atxdad had sent you. Once you are at the right location, 'dir' should show the RUN_ALL file, and you should be good to go.

LegobrandonCP

I get an error message when I type in RUN_ALL.

So I dragged the zip file to my desktop, unzip it, causing it to make a unzipped folder on my desktop.

I then type in 'pwd', 'cd' and 'dir'. When I press enter after 'dir' I get a list of locations on my computer, for example, desktop, documents, pictures, videos and downloads.

What do I do after that?

Poochy

What is that error message though? I'm trying to see if it's because you're not in the correct directory/folder, which seems to be the case.

Sorry for not being clearer about some of the terminologies, as I had assumed you knew what pwd, cd, and dir are for (just goes to show my ages). Those are "DOS prompt" commands that allows you to 'show your current directory', 'change your current directory', and 'list the content of your current directory'.

You want to 'cd' into the directory of the unzipped folder on your desktop. Let's say that is at:

C:\Documents and Settings\Brandon\Desktop\Bricklink

and 'pwd' shows you are at

C:\Documents and Settings\Brandon

then you would need to

cd Desktop\Bricklink

now if you do 'pwd', you should see

C:\Documents and Settings\Brandon\Desktop\Bricklink

and a 'dir' should list the contents of that folder, including the RUN_ALL.m script.

LegobrandonCP

Got it! Thanks for the help @Poochy.

So after all the steps above, I typed in RUN_ALL and a long list of text started showing up. Whats next? Seems to be all error/warning messages.

badbob001

There is an option for US-only vendors or not, but can't there be an option for US-preferred (or country x preferred)? I don't want one international-only piece to cause all my parts to be sourced from International stores.

Poochy

^^ The readme said that you can mostly ignore the warnings. However, you have an error, and the error indicates that it may have something to do with the fact that you're a non-US user (I believe you're in CAN). If so, you may want to read the readme and see what the proposed solution is. I did not run into this issue (or the warning about utf-8 characters).

LegobrandonCP

Poochy said:

^^ The readme said that you can mostly ignore the warnings. However, you have an error, and the error indicates that it may have something to do with the fact that you're a non-US user (I believe you're in CAN). If so, you may want to read the readme and see what the proposed solution is. I did not run into this issue (or the warning about utf-8 characters).

Oops, forgot about that. I will modify the file and update you once I'm done. Again, I really appreciate the help.

LegobrandonCP

Sweet! Got this baby up and running, all thanks to @Poochy. I guess the 'test' set is the Landspeeder. Took a while for a 23 piece set, but its worth it.

EDIT: Just got an error message. :(

Poochy

^ Er, that's a new error to me :(. I have a liboctinterp.dll.a under the Octave\lib\octave\3.6.1\ directory, but don't have the above file (not even sure if this is related or not).

badbob001

How do I know if the script is using the compiled modules or not? When running, mime says "Compiled code available for faster searched". So is it using it or not? And does the multi-cpu option really speed up searching?

atxdad

That message is just a diagnostic to confirm the program's automation has chosen a compiled function among the many delivered that is compatible with the operating system in use. You may assume the compiled code is being used. The parallel processing logic tries to parcel out equal size jobs to every CPU available, so run time roughly follows an inverse law. I declare 8 CPUs on a quad-core hyperthreaded machine and have run times about 1/8th the single CPU time.

badbob001

I'm trying to find a solution to my 130 lot list and it seems to me that once you hit the need to search for a 5 vendor combination, you need to give up unless you have a server farm to play with.

On my 8 core system, a 4 vendor 1.0221e+12 combinations search takes 112 minutes. The 5 vendor 4.5443e+14 combination search looks like will take at least 60 days.

I'm going to check the vendor pruning logic to see if it can give me some hints on the rarest items I should remove from my list. Does it make sense to wish for an option to "Show me a combination of x number of vendors that sells the most (but not all) of the items I want"? When the script is going through the combination search, the progress shows percentages. Is that the percentage of the wanted list that the current vendor combination have? I wouldn't mind getting started with a solution for 80% of my shopping list and not have some rare items hold me back.

Here are some tips on setting up octave:

To have octave always start in the directory with the minimization scripts instead of the user folder:

1. Edit octave\share\octave\site\m\startup\octaverc
2. Find cd(getenv('USERPROFILE')) and add a # in front to comment it out.
   ex:
   #cd(getenv('USERPROFILE'))
   
3. Add a cd(".....") command at the bottom to tell octave to go to a certain directory. You can even have octave go to a network share path (hello server farm...). When using backslash in the path, you must enter it twice \\ because \ by itself has a special meaning.
   ex:
   cd("g:\\bricklink\minimzation_v9")
   cd("\\\\192.168.1.100\\h$\\bricklink\\minimization_v9")
   

To start Octave from a batch script.

start "Octave CPU Any" \\192.168.1.100\h$\bricklink\octave\bin\octave.exe --persist --braindead

To start Octave and run a command. The 'persist' keeps octave running after the command is completed. The 'braindead' suppresses some warnings.

start "Octave CPU Any" \\192.168.1.100\h$\bricklink\octave\bin\octave.exe --persist --braindead --eval "RUN_ALL"

To run Octave on a specific CPU, such as the first core. I like to start the CPU naming count at 0, which makes the later examples less confusing.

start "Octave CPU 0" /affinity 1 \\192.168.1.100\h$\bricklink\octave\bin\octave.exe --persist --braindead --eval "merge_price_guide_data"

To run find_minimum_cost_remote(x) commands in separate CPU cores if you have multiprocessor support enabled in the script.
Putting this in a batch script and running it when prompted by the main script saves a lot of typing. This example is for a 8 core system where the main script in on core 0 and the rest are on cores 1-7.

start "Octave CPU1" /affinity 2 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(1)"

start "Octave CPU2" /affinity 4 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(2)"

start "Octave CPU3" /affinity 8 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(3)"

start "Octave CPU4" /affinity 16 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(4)"

start "Octave CPU5" /affinity 32 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(5)"

start "Octave CPU6" /affinity 64 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(6)"

start "Octave CPU7" /affinity 128 \\192.168.1.100\h$\bricklink\Octave\bin\octave.exe --persist --silent --braindead --eval "find_minimum_cost_remote(7)"

atxdad

@badbob001 A reminder that run time is not so dependent upon the size of your wanted list, but rather the size of your vendor search space. Here are some tips that might help reduce calculation times:

- Enable the vendor pruning tools to reduce the vendor space to only those vendors likely to fill your order. You can prune the vendor list to a size that will run in your allotted time.

- The tools can identify the rarest parts on your wanted list; removing those from your list may allow solutions with few vendors.

- BrickLink's Wanted/ByShop list can already help buyers find vendors meeting most of a wanted list. The purpose of my tool is to help find a 100% solution without having to guess which parts are most important to the user.

- The progress reports during runtime show how much of the vendor search space has been examined, not how much of the wanted list has been filled.

Lastly, thanks for posting the helpful instructions for Octave users.

Titus

I'm curious if anybody knows the solution to this error I'm getting on a 4-vendor search

"Index 80: 81 of 341, 23.75%
error: mxvendorsearchloop: Out of memory."

Thanks