Automated BrickLink cost minimization program available

Tevans333

Can anyone leave some step by step instructions for Octave?

BrunoBen

I have a trouble with ascii but here's my way to install Octave

Can anyone leave some step by step instructions for Octave?

Sorry for my broken english as it's not my native language

Please download octave here
http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.6.1 for Windows MinGW installer/
Choose Octave3.6.1_gcc4.6.2_20120303.7z
Decompress with 7zip or winrar at root
Ex: C:/Octave

Then download octave gui here
http://guioctave.com/?page_id=2
Choose GUIOctave v1.6.0 [7.86Mb]
Launch octavegui
Give it where octave.exe is
For instance c:\Octave\Bin\Octave.exe

You have ready in task bar of GuiOctave

Decompress latest version of scripts in c:\octave\apps for instance

Choose in adress bar the folder
c:\octave\apps

Type RUN_ALL

AvengerDr

hey i just tried your last version and with the default .bsx file it gives me the following error:

Reading wanted list Wanted List for 30053 mini Republic Cruiser.bsx
Requesting data for 1 of 23: Light Bluish Gray Bracket 1 x 2 - 1 x 4
Error using >=
Matrix dimensions must agree.

The only change I made was to change US to EU. This is being run from Matlab r2011b.

Bosstone100

I ran the scripts on a modified .bsx and it looks like it finished. I looked at the Minimum Cost Search Results.txt file and only see the results of the original sample.

Where does the output go exactly?

graphite

^ did you update the input file name in the "parse_wanted_list_and_price_guide.m" file? There is a line in there to change to the name of your input *.bsx file. If you already had that changed try deleting the 2 *.mat files and the minimum cost *.txt file that are the sample outputs.

And the final output should go to a *.txt file as well as it should print in the Octave window.

atxdad

For those following this development, I've recently made the most significant update yet: support for multi-core CPU parallel processing for drastically reduced run times.

As noted earlier, Matlab does not inherently exploit multiple CPU cores which are so common these days, but in my tests, simultaneous Matlab sessions do seem to run on separate cores. Therefore, I automated the division of a large search into smaller jobs that each run in its own Matlab session, and when complete, all results are concatenated into one final report. I can see total CPU utilization of 100% with this modification, and testors confirmed it works with Octave too.

Lastly, for completeness, the search can now support the inclusion of LEGO Pick-A-Brick as a candidate vendor. It requires some manual data entry to use, but it's there now for those seeking a true least-cost solution.

Thanks for the continued suggestions and especially the code samples for me to copy!

Bosstone100

@atxdad - do we have to split up the .bsx into four files or will all session work together to find the final result?

atxdad

Nope. Just make one wanted list. The program divides the search labor among as many cores as you wish to throw at the problem. I tested up to 8, but it shouldn't have any limit.

graphite

It is a very nice improvement. The 4 vendor searches that I've done that would normally take probably 6-8 hours or more take about 3 running on 7 cores. Would be less probably but I think atxdad is still working on balancing the work when it is split up.

effalcon

very nice update.

Coolsplash

Can anyone help me walk through the installation for this script/program?

Tevans333

I'm having trouble getting multicores to run in Octave.

The script prompts me the following:

Multi_core CPU processing enabled. Instructions:
1) Start 2 additional Matlab session(s) if not already running
2) Ensure all working directories are hte same as this sessions
3) Run the following 2 command(s) in the other 2 session(s)
In session 1 run: find_minimum_cost_remote(1)
In session 2 run: find_minimum_cost_remote(2)

I'm good up until step 3. Whenever I try to run the command it tells me the script doesn't exist. Can someone please let me know how the command should look? I'm typing "run find_minimum_cost_remote(1)"

atxdad

@Tevans333 - Although I use Matlab, here are some things to try; Octave users can correct me if these aren't valid commands:

1) Verify that the files 'find_minimum_cost.m' and 'find_minimum_cost_remote.m'
both exist in the same directory

2) In the two additional Octave sessions, do this:
a) Use the 'cd' command to point the workspace to the directory verified above
b) Use the 'dir' or 'ls' command to verify the scripts are in your current working directory
c) Run these commands:
which find_minimum_cost.m
which find_minimum_cost_remote.m
They should both list the current working directory

If any of those verifications fail, PM me the screen messages and we'll try some more.
Maybe an Octave user can confirm that you have launched the functions (not scripts) with the correct syntax.

graphite

I'm not sure why but for some reason you don't have to type 'run ' in front of the find_minimum_cost_remote(x) calls in Octave. That is probably what is causing it not to work for @Tevans333. I think it has to do with it being a "function" or "script" instead of a "program" but @atxdad would know the difference between 'find_minimum_cost' and 'find_minimum_cost_remote' from a programming standpoint not functionality.

Tevans333

Thanks guys for the quick reposnes.

@graphite, you are right, I was typing run and I didn't need to be. Now I'm happily running @atxdad's scripts on multi cores:)

Tevans333

Ok, next issue I could use some help on:)

After running it on 4 cores it completed a 4 vendor, 33 lot search in about 2.5 hours, very happy with this! But at the end it did not save to the results to the "Minimum cost search results.txt" file. The script prompted me to wait untill all sessions had completed. I did this and made sure each had a command prompt. I then hit a key to continue and have the results saved, only problem is the results file only has this:

Checking 1.06133e+009 combinations of 4 vendors for those that have desired quantities...

Search completed for this session. Inspect progress of other Matlab sessions
and press any key to continue only when other results are ready. PAUSED
Loading results: multiporc_output_1.mat
Loading results: multiporc_output_2.mat
Loading results: multiporc_output_3.mat

I made sure each session was working in the same directory. Is there a way to generate the results without running the 2.5 hour search again? What may have gone wrong?

atxdad

@Tevans333 - The likely answer is that there were no solutions found, and with nothing to report, the program just stops. I've since added some debug text to make that clear, i.e. "This page intentionally left blank." I'll PM you with a few more questions soon to verify this is what happened.

This is probably a good time to say that any run-time questions like this can be sent to me directly. Remote debugging is sometimes an iterative process, and I'm guessing few would be interested in my whole conversation.

caperberry

Hi guys, I wanted to ask a big favour - would anyone be prepared to run a report for me in this program? I don't have Matlab and need to cost up a model for a charity. It has approx 130 lots.

I can supply the parts list as BrickLink XML, .bsx, .ldr or .txt

Any help gratefully received!

caperberry

oops double post

atxdad

Sure, @caperberry, I'll run it for you. I'll PM you with details of what I need.

Mad_Dog

I have been using the program for a few weeks and for me works quite good, I am open to any improvements on it.

Running list of 115 batches, 3091 pieces, in 2 different files of 37 batches / 837 pieces and 78 / 2254 goes more or less OK with default options, of course 4 shops is looooong calculation, and not always successful, other searches are much better, so keep low batches and merge after, that is the best option.

Atxdad, I am happy to keep checking any new version you may need, keep the good work.

Rgrds

J.

caperberry

OK I can report back that this is great. In short, I'm saving about £15-20 on an order that I reckon would have cost £65-70 had I just used my normal route of trying to do some manual analysis of the BL Wanted by Store results.

The parts in my order were not expensive but I had 140 different lots, which meant I had to buy from at least 3 sellers. But it turned out buying from 4 vendors was cheaper, even after adding (estimated) postage manually (which of course cannot be calculated automatically).

As the program cannot limit by country/territory, I went through the many output options provided and rejected non-European stores; there were still several options for me to choose from.

The program helped me realise I would have to compromise in some way, which I did by having some Used parts instead. Not an easy decision, but great to be able to make it fully informed.

Now using BrickLink the usual way is just depressing! - so much effort just to know that you are getting a decent price, and it feels like BL is heavily weighted in favour of the seller. A program like this is the way forward and I hope @atxdad has continued success developing this and getting it out there. If BL wants to stay successful they'd be wise to integrate this program into the site, before someone else just does it elsewhere...

Bosstone100

It really is an amazing program.

Tevans333

BEWARE!!! This program is costing me a ton more money than before I started using it...because it makes projects so much easier and cheaper I find myself doing more projects:)

caperberry

^hahahaha

Molican

Can anyone send me GUIOctave v1.6.0 because the site is down since at least some days? PM me for my email.
Thanks
M.

samiam391

The program doesn't seem to be working for me. Every time I click "StartLDDManager" nothing happens at all. I've installed the Access Runtime, so that's not the problem. Any suggestions/tips? Hope I'm on topic here... Right now, I really want any sort of automated system that will upload my parts to BL, so if there's something better than LDDmanager, please let me know!

graphite

did you read the readme.txt? Specifically "For Access 2007 and 2010 users - for LDDManager to work you may need to add it to "Trusted Locations".
Read here how to do that: http://www.btabdevelopment.com/ts/default.aspx?PageId=13"

pilfos

Hello,

I've changed the numproc value to 8 in find_minimum_cost.m but I don't know how to open new sessions of Octave.

And now, I'm receiving this error:

>> RUN_ALL
Reading wanted list 10179-bb.bsx
Requesting data for 1 of 41: Light Bluish Gray Technic, Pin Connector Round (Pin Joiner Round)
warning: E:\Dev\Octave\apps\readpriceguide.m: possible Matlab-style short-circuit operator at line 63, column 32
error: findnextstring: mx_el_ge: nonconformant arguments (op1 is 1x41, op2 is 0x0)
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: E:\Dev\Octave\apps\findnextstring.m at line 7, column 6
error: E:\Dev\Octave\apps\parsepriceguidestring.m at line 33, column 14
error: E:\Dev\Octave\apps\readpriceguide.m at line 52, column 92
error: E:\Dev\Octave\apps\parse_wanted_list_and_price_guide.m at line 80, column 58
error: E:\Dev\Octave\apps\RUN_ALL.m at line 13, column 1
>>

Thanks!

atxdad

@pilfos It's my experience so far that almost all run-time errors in the parsing scripts are due to unexpected currency types. If you are using the scripts outside of the US, check the notes for international users in the README, otherwise send me a PM and we'll try to debug further.

graphite

As far as opening multiple instances of Octave, first do you have a multi-processor/multi-core system? As far as windows goes, just run the program (click the shortcut) for as many you want to open. Once you have the 8 open you need to open the Task Manager (right click in the open space on the start menu/task bar and click 'Start Task Manager'). Sort by process name, you should see 8 octave.exe's listed. Right click on each one, go to Set Affinity, and select a different core for each one. Then continue to follow atxdad's directions for running multiple processes.

pilfos

yes, the needed files were modified for taking into account the EUR, and the script went fine a week ago. But yesterday, when I was trying to check the multicore option, I started to get this error.

I have an Intel i7, so I think I can use 8 cores, or only 4?

But now Bricklink is down, so I can't test your suggestions.

Thnks!

dustinmroberts

Anybody have any luck using this after all the Bricklink problems? I tried to run a file through and all that was returned in the Minimum cost search results file was this:

Checking 724 combinations of 1 vendors for those that have desired quantities...
Checking 261726 combinations of 2 vendors for those that have desired quantities...
Checking 62988724 combinations of 3 vendors for those that have desired quantities...


I've run it a few times changing the maximum number of stores each time and still the same results.

atxdad

Knowing of BrickLink's changes, I've been checking functionality of the parser every few days, and it seems to still be working correctly as of now. I'll keep an eye on it during this transition period.

@dustinmroberts Did the program run to completion when it created that text file output, and are you using the latest version (V5.zip)? In older versions, that output file would mean no solutions were found, so it had nothing to report. More recent versions explicitly say 'No combinations found' to avoid confusion. PM me with any replies and we'll figure it out.

kuf

I have a trouble with ascii but here's my way to install Octave

Can anyone leave some step by step instructions for Octave?

Sorry for my broken english as it's not my native language

Please download octave here
http://sourceforge.net/projects/octave/files/Octave Windows binaries/Octave 3.6.1 for Windows MinGW installer/
Choose Octave3.6.1_gcc4.6.2_20120303.7z
Decompress with 7zip or winrar at root
Ex: C:/Octave

Then download octave gui here
http://guioctave.com/?page_id=2
Choose GUIOctave v1.6.0 [7.86Mb]
Launch octavegui
Give it where octave.exe is
For instance c:\Octave\Bin\Octave.exe

You have ready in task bar of GuiOctave

Decompress latest version of scripts in c:\octave\apps for instance

Choose in adress bar the folder
c:\octave\apps

Type RUN_ALL

Just so you know, the site guioctave.com/?page_id=2 has been in "Maintenance Mode" since at least May 20th, from what I read searching around. You'll need to use this link: dl.dropbox.com/u/20202519/GUIOctave_1.6.0.exe

kuf

Has anybody made a master file of all bricks available from LEGO Pick a Brick? (longshot, I know)

DadsAFOL

Has anybody made a master file of all bricks available from LEGO Pick a Brick? (longshot, I know)

I started, but man is it a pain. I got about 10% done and it took an hour. I don't have 9 more hours to give to it.

I would be willing to pay someone $25 for the PaB bsx file. So all you entrepreneurs, listen up!

kuf

I was thinking about filtering by color then copying all items over to excel and trying to do a VLookup with descriptions somehow. It's too bad the descriptions don't have the item numbers in them.
My other idea was contacting TLG and asking them for an excel file of the info, explaining we wanted it so we could consider choosing PAB as a vendor option.

brickmatic

^ and ^^ So you guys know what the PaB bricks are, you just need to put it into a bsx file?

maques

I think this could be done very easily* [and I will make one eventually...]

I've checked PAB already, the item's could be extracted easily from the site, eg:
http://customization.lego.com/en-US/pab/service/getBricks.aspx?st=5&sv=allbricks&pn=0&ps=500&cat=US
and then query all brick details by the appropriate query, eg.:
http://customization.lego.com/en-US/pab/service/getBrick.aspx?itemid=4211050&st=5&sv=allbricks&pageNo=0&pageSize=10&cat=US

*Problems/issues encountered so far:

- This is not "_A_ shop", so price and currency is different from the "cat=US" vs "cat=DE" (and so on).


- meanwhile basic bricks match the BL's ID's, like

LEGO: Element ID: 4211050 - 1 1/2 M Connecting Bush
Design ID: 32002
Exact Color: DARK STONE GREY

would "translate" to -->

Dark Bluish Gray Technic, Pin 3/4
http://www.bricklink.com/catalogPG.asp?P=32002&colorID=85

but:

LEGO: Element ID: 4275585 - Mini Upper Part "No. 1197"
Design ID: 76382
Exact Color: WHITE

cannot be automatically translated to:

White Torso Town Prisoner Number 50380, Dark Bluish Gray Stripes, Buttons, Hairy Chest Pattern / White Arms / Yellow Hands
http://www.bricklink.com/catalogPG.asp?P=973px432c01&colorID=1

This applies to most of the minifig parts I guess...


- I would need a color translation table, which maps LEGO color names to BL color ID-s. This is also not a big task to make, but I haven't completed that yet. If anyone has this already, post it or pm me. Currently available PAB colors [at some I put the BL color ID]... Fill the gap :-)

#BL_color_ID color_family "exact color [with spaces]"
11 Black BLACK
? Blue BRIGHT BLUE
? Blue EARTH BLUE
? Blue MEDIUM BLUE
? Blue TR. BLUE
? Blue TR. LIGHT BLUE
? Green BRIGHT GREEN
? Green BRIGHT YELLOWISH GREEN
? Green DARK GREEN
? Green TR. FLUORE. GREEN
? Green TR. GREEN
? Grey DARK STONE GREY
? Grey MEDIUM STONE GREY
? Orange-brown BRIGHT ORANGE
? Orange-brown DARK ORANGE
? Orange-brown REDDISH BROWN
? Orange-brown TR. BROWN
? Purple BRIGHT PURPLE
? Purple LIGHT PURPLE
? Purple MEDIUM LILAC
? Red BRIGHT RED
? Red NEW DARK RED
? Red TR. FLUORE.REDD. ORANGE
? Red TR. RED
1 White WHITE
3 Yellow BRICK YELLOW
? Yellow BRIGHT YELLOW
2 Yellow SAND YELLOW
? Yellow TR. YELLOW


- Decide what to do with "unavailable" items. Eg. item exists, but no stock.


- Updates...

maques

Thread seems to be died... Anyone still interested?

I did some work on it, beta results here:
http://maques.hu/lego/auto-pab/

pablist-us.txt has 1532 entries, most of them matched easily with BrickLink, though I added some manual item/group translation tables.

Color translation table is there too, could be some errors in both the item and the color table.

80 items left on "todo" list to be manually looked up on BL - 99% minifig parts.

Could look up other country prices (UK, DE, HU, PL ...)

Could look up BL prices [1/lots for US/EU etc sellers]

Could create csv or any different format of the result

atxdad

@maques : The script readLEGOPAB.m expects a .BSX format file (text file with markups), but it only parses out a few of the fields that normally exist in a .BSX. The critical information it needs (part number, color, price) are found in your pablist-us.txt.

I can write a script that will create a .bsx text file, putting the expected markups around the values you've pulled. I can guarantee to make it readable by readLEGOPAB.m, but it may take some extra work to get it to import to BrickStore if one wanted to visually verify its contents or manually make an edit.

Is pablist-us.txt the format of your final product, and is there any opportunity for revision? Also note that BL prices will not be necessary, nor will a .csv format. PM me and we can finalize everything. Thanks for the help.

DadsAFOL

@maques The bsx file you created is excellent. There were just a few items that came up as "undertermined" and needed the "b" added to the part number. That was easily done in BrickStore. The handful of remaining items that didn't link can be added manually in BrickStore.

Thanks for doing this!

maques

Can you let me know what are the items needed to be modified?
So in case PAB item updates or for any other reason I shall create a new bsx then they would be corrected and wouldn't need to have modified by "end-user" :-)

DadsAFOL

The unidentified were 1x1, 1x2, 2x2 tiles, 1x2 mod tile with grille, and a couple others. In BrickStore, just type "undertermined" in the filter list at the top and you will see them. I didn't go through the 80 unmatched yet.

atxdad

With the PAB inventory importation now getting some attention, an error in my scripts during the reporting of PAB information was found which could cause some runtime confusion but not affect end results. Anyone using the PAB feature can PM me to request a hotfix immediately.

I am planning to release another revision update soon which will include this fix and few others, but I think I will wait until the work of @maques is complete to ensure compatibility. The draft .bsx file he has already provided is a huge timesaver.

maques

I made some adjustments, so now bsx contains:
- qty 9999 for item on stock and qty 0 for N/A stock item
- removed "fixme/error" items - so no ~80 warnings with BrickStore
- All LEGO/PAB information in Remarks field

Current issues/todo:
- 1 unknown item stayed with BrickStore, but I don't yet had time to check why, as item 50861 in color 11 [black] exist.
[Catalog: Parts: Tire & Tread: 50861 - Tire 21mm D. x 6mm City Motorcycle]

- The missing/to be checked-and-fixed 86 items are in:
http://maques.hu/lego/auto-pab/bltranslate-todo.txt
even if people here can lookup and match 1-1 item and post details here (eg what LEGO PAB Element # is the matching BL item #), then it would be helpful

- The "undetermined" elements come from EXACT element matching, like 3069/3070, where it comes from LEGO/PAB Design ID being 3069/3070.
Because for 3069b/3070b the PAB Design ID-s 30070/30039. Since there is no way for me to find out wheter a LEGO 3069/3070 is without or with a groove, so type A or B, I can automatch group (or single item).
After I automatch 3068/3069/3070/2412, there are still around 16 other items stay in undetermined state, eg. with undetermined stud, hinge, handle, ends&attachments, back and hole types.
What to do with them?

Anything else?

maques

Was wondering...:

Wouldn't it be good/useful/interesting to set up a "phantom" Bricklink Store with the PAB items, so could be updated and matched wanted lists against it?
Then there could be different ones for countries as the PAB USD/EUR/GBP/HUF/PLN/HUF etc prices are static...

Problem would be that these shops would come out all the time but you wouldn't be able to order as it would be just a reference shop...

So it would need BL administration + programming to make these hidden / available per request and also not to count in the ~1500x9999 =~ 15 million extra parts...

maques

http://maques.hu/lego/auto-pab/pablist-us-b.bsx
is with automatch groups 3068/3069/3070/2412 to "b" (with groove/lip)

Still don't understand the unknown motorcycle tire. BrickStore's database out of sync, or not/can't do lookup to BL?

dustinmroberts

Is there currently a way to exclude a particular shop? I keep having a shop show up that only has 1-2 parts, but they have a minimum buy that is more expensive than the parts I would need to purchase.