What should be the best way to open a text file and compare a line input for duplication and if it is duplicated delete the duplicated line from the text file. Trying to find some information on that but nothing specific.
Thanks for helping!
For small files read the file a line at a time and add the lines to a hash.
When you have read a line first check that it's not in the hash already. If not, print it to the output file and insert it into the hash, otherwise skip to the next line.
If you can't figure that out, write some code that is your best guess at how to do it, ask again and include your code.
DWIM is Perl's answer to Gödel
tag1:xxxxxxx tag2:xxxxxx tag3:xxxxxxx tag4:yyyyy
How can I remove all lines that have the same tags 1 through 3 and replace it with a single line that has a new tag4? Currently I am able to remove all excess lines with the same tags 1 through 3 using your method but am unable to change tag4 because the hash method works by not writing subsequent values. Hence once I find out I have a duplicate it is too late to change it as the first has already been written. Any suggestions? Thanks!
Provide half a dozen lines of sample data, the test code you are currently using, and a sample of the output you expect to see.
For the test code it is easiest to use a __DATA__ section for the test data rather than an external file and simply print the result rather than generating an external file.
DWIM is Perl's answer to Gödel
My data would looks like this:
MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0
MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA1
so every set of data (MCAT, PCAT, LMAT) is unique except for the fourth and eight which differ only by TYPE.
I would like to roll the fourth and eight together so it would look like this:
MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0, KA1
MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
This leaves me with a total of 7 data groups instead of 8.
So far my code doesn't even come close to accomplishing doing this....
The following is ok for reasonable size files but may bog down when things get huge.
use strict;
use warnings;
use Data::Dump::Streamer;
my %firstLines;
my @lines;
while () {
chomp;
my ($data, $type) = /(.*)\s+TYPE:\s+(\w+)$/;
next if ! defined $type; # ignore malformed line
if (exists $firstLines{$data}) {
$lines[$firstLines{$data}] .= ", $type";
} else {
$firstLines{$data} = @lines;
push @lines, $_;
}
}
print join "\n", @lines;
__DATA__
MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0
MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0
MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1
MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA1
Prints:
MCAT: 0xf30cbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0 MCAT: 0xcc3fbed1 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1 MCAT: 0xeeccbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1 MCAT: 0xf30cbe91 PCAT: 0xafaddd09 LMAT: 0x00040000 TYPE: KA0, KA1 MCAT: 0xeeecbe01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA0 MCAT: 0xcc000331 PCAT: 0x000fb109 LMAT: 0x00000800 TYPE: KA1 MCAT: 0xe554be01 PCAT: 0xcda2b409 LMAT: 0x00100000 TYPE: KA1
DWIM is Perl's answer to Gödel
Thank you very much!!! I really appreciate you solving that problem for me. I'm very new to PERL and have to admit that your code took a while to make sense to me. I didn't realize that you could access an array by the data value, I thought you had to access it by location (0,1,2,3... etc). Thanks again for your help!
Just in case there is some confusion or misunderstanding of some of the Perl tricks used I better go through some of that code and elaborate on what's happening. Note that I've taken interesting lines in processing order rather that the order they are coded.
$firstLines{$data} = @lines;this is a little tricksy. It creates a new entry in %firstLines that contains the index to the new line as the value and is keyed by the unique part of the line contents. @lines in scalar context returns the number of elements in the array.
if (exists $firstLines{$data})checks to see if we've already seen a specific line.
$lines[$firstLines{$data}] .= ", $type";builds the multiple entries for duplicated lines. Note that $firstLines{$data} returns the index number that was stored earlier.
DWIM is Perl's answer to Gödel
sort -u infile > outfile # or (you might many some of uniq's extra options): sort infile | uniq > outfile
my $last = $_ = <>;
print;
while (<>)
{
print if ($_ ne $last);
$last = $_;
}
Also note that, of the solutions provided thus far, the hash-based option is the only one which will both eliminate all duplicates (printing only the first appearance of each line) and also preserve the original order of the (remaining) lines, which may or may not be significant to you.